Package net.snowflake.ingest.streaming

Interface SnowflakeStreamingIngestChannel

public interface SnowflakeStreamingIngestChannel

A logical partition that represents a connection to a single Snowflake table, data will be ingested into the channel, and then flushed to Snowflake table periodically in the background.

Channels are identified by their name and only one channel with the same name may ingest data at the same time. When a new channel is opened, all previously opened channels with the same name are invalidated (this applies for the table globally. not just in a single JVM). In order to ingest data from multiple threads/clients/applications, we recommend opening multiple channels, each with a different name.

Thread safety note: Implementations of this interface are required to be thread safe.

Method Summary

Modifier and Type	Method	Description
CompletableFuture<Void>	close()	Close the channel, this function will make sure all the data in this channel is committed
CompletableFuture<Void>	close(boolean drop)	Close the channel, this function will make sure all the data in this channel is committed
String	getDBName()	Get the database name
String	getFullyQualifiedName()	Get the fully qualified channel name
String	getFullyQualifiedTableName()	Get the fully qualified table name that the channel belongs to
String	getLatestCommittedOffsetToken()	Get the latest committed offset token from Snowflake
String	getName()	Get the name of the channel
String	getSchemaName()	Get the schema name
String	getTableName()	Get the table name
Map<String,ColumnProperties>	getTableSchema()	Gets the table schema associated with this channel.
InsertValidationResponse	insertRow(Map<String,Object> row, String offsetToken)	Insert one row into the channel, the row is represented using Map where the key is column name and the value is a row of data.
InsertValidationResponse	insertRows(Iterable<Map<String,Object>> rows, String offsetToken)	Insert a batch of rows into the channel with the end offset token only, please see insertRows(Iterable, String, String) for more information.
InsertValidationResponse	insertRows(Iterable<Map<String,Object>> rows, String startOffsetToken, String endOffsetToken)	Insert a batch of rows into the channel, each row is represented using Map where the key is column name and the value is a row of data.
boolean	isClosed()
boolean	isValid()

Method Detail

getFullyQualifiedName

String getFullyQualifiedName()

Get the fully qualified channel name

Returns:

fully qualified name of the channel, in the format of dbName.schemaName.tableName.channelName

getName

String getName()

Get the name of the channel

Returns:

name of the channel

getDBName

String getDBName()

Get the database name

Returns:

name of the database

getSchemaName

String getSchemaName()

Get the schema name

Returns:

name of the schema

getTableName

String getTableName()

Get the table name

Returns:

name of the table

getFullyQualifiedTableName

String getFullyQualifiedTableName()

Get the fully qualified table name that the channel belongs to

Returns:

fully qualified table name, in the format of dbName.schemaName.tableName

isValid

boolean isValid()

Returns:

a boolean which indicates whether the channel is valid

isClosed

boolean isClosed()

Returns:

a boolean which indicates whether the channel is closed

close

CompletableFuture<Void> close()

Close the channel, this function will make sure all the data in this channel is committed

Returns:

a completable future which will be completed when the channel is closed

close

CompletableFuture<Void> close(boolean drop)

Close the channel, this function will make sure all the data in this channel is committed

Note that this call with drop=true will delete Offset Token and other state from Snowflake servers unless the channel has already been opened in another client. So only use it if you are completely done ingesting data for this channel. If you open a channel with the same name in the future, it will behave like a new channel.

Parameters:

drop - if true, the channel will be dropped after all data is successfully committed.

Returns:

a completable future which will be completed when the channel is closed

insertRow

InsertValidationResponse insertRow(Map<String,Object> row,
                                   @Nullable
                                   String offsetToken)

Insert one row into the channel, the row is represented using Map where the key is column name and the value is a row of data. The following table summarizes supported value types and their formats:

Snowflake Column Type	Allowed Java Data Type
CHAR, VARCHAR	String primitive data types (int, boolean, char, …)
BINARY	byte[] String (hex-encoded)
NUMBER, FLOAT	numeric types (BigInteger, BigDecimal, byte, int, double, …) String
BOOLEAN	boolean numeric types (BigInteger, BigDecimal, byte, int, double, …) String See boolean conversion details.
TIME	LocalTime OffsetTime String (in one of the following formats): DateTimeFormatter.ISO_LOCAL_TIME DateTimeFormatter.ISO_OFFSET_TIME Integer-stored time (see Snowflake Docs for more details)
DATE	LocalDate LocalDateTime OffsetDateTime ZonedDateTime Instant String (in one of the following formats): DateTimeFormatter.ISO_LOCAL_DATE DateTimeFormatter.ISO_LOCAL_DATE_TIME DateTimeFormatter.ISO_OFFSET_DATE_TIME DateTimeFormatter.ISO_ZONED_DATE_TIME Integer-stored date (see Snowflake Docs for more details)
TIMESTAMP_NTZ, TIMESTAMP_LTZ, TIMESTAMP_TZ	LocalDate LocalDateTime OffsetDateTime ZonedDateTime Instant String (in one of the following formats): DateTimeFormatter.ISO_LOCAL_DATE DateTimeFormatter.ISO_LOCAL_DATE_TIME DateTimeFormatter.ISO_OFFSET_DATE_TIME DateTimeFormatter.ISO_ZONED_DATE_TIME Integer-stored timestamp (see Snowflake Docs for more details) For TIMESTAMP_LTZ and TIMESTAMP_TZ, all input without timezone will be by default interpreted in the timezone "America/Los_Angeles". This can be changed by calling OpenChannelRequest.OpenChannelRequestBuilder.setDefaultTimezone(ZoneId).
VARIANT, ARRAY	String (must be a valid JSON value) primitive data types and their arrays BigInteger, BigDecimal LocalDate LocalDateTime OffsetDateTime ZonedDateTime Map<String, T> where T is a valid VARIANT type T[] where T is a valid VARIANT type List<T> where T is a valid VARIANT type
OBJECT	String (must be a valid JSON object) Map<String, T> where T is a valid variant type
GEOGRAPHY, GEOMETRY	Not supported

Parameters:

row - object data to write. For predictable results, we recommend not to concurrently modify the input row data.

offsetToken - offset of given row, used for replay in case of failures. It could be null if you don't plan on replaying or can't replay

Returns:

insert response that possibly contains errors because of insertion failures

insertRows

InsertValidationResponse insertRows(Iterable<Map<String,Object>> rows,
                                    @Nullable
                                    String startOffsetToken,
                                    @Nullable
                                    String endOffsetToken)

Insert a batch of rows into the channel, each row is represented using Map where the key is column name and the value is a row of data. See insertRow(Map, String) for more information about accepted values.

Parameters:

rows - object data to write

startOffsetToken - start offset of the batch/row-set

endOffsetToken - end offset of the batch/row-set, used for replay in case of failures, * It could be null if you don't plan on replaying or can't replay

Returns:

insert response that possibly contains errors because of insertion failures

insertRows

InsertValidationResponse insertRows(Iterable<Map<String,Object>> rows,
                                    @Nullable
                                    String offsetToken)

Insert a batch of rows into the channel with the end offset token only, please see insertRows(Iterable, String, String) for more information.

getLatestCommittedOffsetToken

@Nullable
String getLatestCommittedOffsetToken()

Get the latest committed offset token from Snowflake

Returns:

the latest committed offset token

getTableSchema

Map<String,ColumnProperties> getTableSchema()

Gets the table schema associated with this channel. Note that this is the table schema at the time of a channel open event. The schema may be changed on the Snowflake side in which case this will continue to show an old schema version until the channel is re-opened.

Returns:

map representing Column Name to Column Properties