net.snowflake.ingest

Class SimpleIngestManager

java.lang.Object

net.snowflake.ingest.SimpleIngestManager

All Implemented Interfaces:

AutoCloseable

public class SimpleIngestManager
extends Object
implements AutoCloseable

This class provides a basic, low-level abstraction over the Snowflake Ingest Service REST api

Usage of this class delegates all exception and state handling to the developer

Author:

obabarinsa

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	SimpleIngestManager.Builder This Builder allows someone to configure a SimpleIngestManager prior to instantiating the manager

Constructor Summary

Constructors

Constructor and Description
SimpleIngestManager(String account, String user, String pipe, KeyPair keyPair) Deprecated.
SimpleIngestManager(String account, String user, String pipe, KeyPair keyPair, String schemeName, String hostName, int port) Deprecated.
SimpleIngestManager(String account, String user, String pipe, PrivateKey privateKey) Constructs a SimpleIngestManager for a given user in a specific account In addition, this also takes takes the target table and source stage Finally, it also requires a valid private key registered with Snowflake DB
SimpleIngestManager(String account, String user, String pipe, PrivateKey privateKey, String schemeName, String hostName, int port) Constructs a SimpleIngestManager for a given user in a specific account In addition, this also takes takes the target table and source stage Finally, it also requires a valid private key registered with Snowflake DB
SimpleIngestManager(String account, String user, String pipe, PrivateKey privateKey, String schemeName, String hostName, int port, String userAgentSuffix)
SimpleIngestManager(String account, String user, String pipe, String hostName, KeyPair keyPair, String userAgentSuffix) Using this constructor for Builder pattern.

Method Summary

Modifier and Type	Method and Description
void	close() Closes the resources associated with this object.
ConfigureClientResponse	configureClient(UUID requestId) Register a snowpipe client and returns the client sequencer
String	getAccount() getAccount - Gives back the name of the account that this IngestManager is targeting
ClientStatusResponse	getClientStatus(UUID requestId) Get client status for snowpipe which contains offset token and client sequencer
HistoryResponse	getHistory(UUID requestId, Integer recentSeconds, String beginMark) Pings the service to see the current ingest history for this table
HistoryRangeResponse	getHistoryRange(UUID requestId, String startTimeInclusive, String endTimeExclusive) Pings the service to see the current ingest history for this table
String	getPipe() getPipe - gives back the pipe which we are using
RequestBuilder	getRequestBuilder()
String	getUser() getUser - gives back the user on behalf of which this ingest manager is loading
IngestResponse	ingestFile(StagedFileWrapper file, UUID requestId) ingestFile - ingest a single file
IngestResponse	ingestFile(StagedFileWrapper file, UUID requestId, boolean showSkippedFiles) ingestFile - ingest a single file
IngestResponse	ingestFiles(List<StagedFileWrapper> files, UUID requestId) ingestFiles - synchronously sends a request to the ingest service to enqueue these files
IngestResponse	ingestFiles(List<StagedFileWrapper> files, UUID requestId, boolean showSkippedFiles) ingestFiles - synchronously sends a request to the ingest service to enqueue these files
IngestResponse	ingestFiles(List<StagedFileWrapper> files, UUID requestId, boolean showSkippedFiles, InsertFilesClientInfo clientInfo) ingestFiles With Client Info - synchronously sends a request to the ingest service to enqueue these files along with clientSequencer and offSetToken.
static List<StagedFileWrapper>	wrapFilepaths(Set<String> filenames) wrapFilepaths - convenience method to take a list of filenames and produce a list of FileWrappers with unset size

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SimpleIngestManager

@Deprecated
public SimpleIngestManager(String account,
                                       String user,
                                       String pipe,
                                       KeyPair keyPair)

Deprecated.

Constructs a SimpleIngestManager for a given user in a specific account In addition, this also takes takes the target table and source stage Finally, it also requires a valid KeyPair object registered with Snowflake DB

This method is deprecated, please use the constructor that only requires PrivateKey instead of KeyPair.

Parameters:

account - The account into which we're loading Note: account should not include region or cloud provider info. e.g. if host is testaccount.us-east-1.azure .snowflakecomputing.com, account should be testaccount

user - the user performing this load

pipe - the fully qualified name of the pipe

keyPair - the KeyPair we'll use to sign JWT tokens

SimpleIngestManager

@Deprecated
public SimpleIngestManager(String account,
                                       String user,
                                       String pipe,
                                       KeyPair keyPair,
                                       String schemeName,
                                       String hostName,
                                       int port)

Deprecated.

Constructs a SimpleIngestManager for a given user in a specific account In addition, this also takes takes the target table and source stage Finally, it also requires a valid KeyPair object registered with Snowflake DB

This method is deprecated, please use the constructor that only requires PrivateKey instead of KeyPair.

Parameters:

account - the account into which we're loading Note: account should not include region or cloud provider info. e.g. if host is testaccount.us-east-1.azure .snowflakecomputing.com account should be testaccount If this is the case, you should use the constructor that accepts hostname as argument

user - the user performing this load

pipe - the fully qualified name of the pipe

keyPair - the KeyPair we'll use to sign JWT tokens

schemeName - http or https

hostName - the hostname

port - the port number

SimpleIngestManager

public SimpleIngestManager(String account,
                           String user,
                           String pipe,
                           PrivateKey privateKey)
                    throws InvalidKeySpecException,
                           NoSuchAlgorithmException

Constructs a SimpleIngestManager for a given user in a specific account In addition, this also takes takes the target table and source stage Finally, it also requires a valid private key registered with Snowflake DB

Note: this method only takes in account parameter and derive the hostname, i.e. testaccount.snowfakecomputing.com. If your deployment is not aws us-west, please use the constructor that accept hostname as argument

Parameters:

account - The account into which we're loading Note: account should not include region or cloud provider info. e.g. if host is testaccount.us-east-1.azure .snowflakecomputing.com, account should be testaccount. If this is the case, you should use the constructor that accepts hostname as argument

user - the user performing this load

pipe - the fully qualified name of the pipe

privateKey - the private key we'll use to sign JWT tokens

Throws:

NoSuchAlgorithmException - if can't create key factory by using RSA algorithm

InvalidKeySpecException - if private key or public key is invalid

SimpleIngestManager

public SimpleIngestManager(String account,
                           String user,
                           String pipe,
                           String hostName,
                           KeyPair keyPair,
                           String userAgentSuffix)

Using this constructor for Builder pattern. KeyPair can be passed in now since we have made @see Utils.createKeyPairFromPrivateKey(PrivateKey) public

Parameters:

account - The account into which we're loading Note: account should not include region or cloud provider info. e.g. if host is testaccount.us-east-1.azure .snowflakecomputing.com, account should be testaccount. If this is the case, you should use the constructor that accepts hostname as argument

user - the user performing this load

pipe - the fully qualified name of the pipe

hostName - the hostname

keyPair - keyPair associated with the private key used for authentication. See @see Utils.createKeyPairFromPrivateKey(java.security.PrivateKey) to generate KP from p8Key

userAgentSuffix - user agent suffix we want to add.

SimpleIngestManager

public SimpleIngestManager(String account,
                           String user,
                           String pipe,
                           PrivateKey privateKey,
                           String schemeName,
                           String hostName,
                           int port)
                    throws NoSuchAlgorithmException,
                           InvalidKeySpecException

Constructs a SimpleIngestManager for a given user in a specific account In addition, this also takes takes the target table and source stage Finally, it also requires a valid private key registered with Snowflake DB

Parameters:

account - the account into which we're loading Note: account should not include region or cloud provider info. e.g. if host is testaccount.us-east-1.azure .snowflakecomputing.com, account should be testaccount

user - the user performing this load

pipe - the fully qualified name of the pipe

privateKey - the private key we'll use to sign JWT tokens

schemeName - http or https

hostName - the hostname i.e. testaccount.us-east-1.azure .snowflakecomputing.com

port - the port number

Throws:

NoSuchAlgorithmException - if can't create key factory by using RSA algorithm

InvalidKeySpecException - if private key or public key is invalid

SimpleIngestManager

public SimpleIngestManager(String account,
                           String user,
                           String pipe,
                           PrivateKey privateKey,
                           String schemeName,
                           String hostName,
                           int port,
                           String userAgentSuffix)
                    throws NoSuchAlgorithmException,
                           InvalidKeySpecException

Throws:

NoSuchAlgorithmException

InvalidKeySpecException

Method Detail

getAccount

public String getAccount()

getAccount - Gives back the name of the account that this IngestManager is targeting

Returns:

the name of the account

getUser

public String getUser()

getUser - gives back the user on behalf of which this ingest manager is loading

Returns:

the user name

getPipe

public String getPipe()

getPipe - gives back the pipe which we are using

Returns:

the pipe name

wrapFilepaths

public static List<StagedFileWrapper> wrapFilepaths(Set<String> filenames)

wrapFilepaths - convenience method to take a list of filenames and produce a list of FileWrappers with unset size

Parameters:

filenames - the filenames you want to wrap up

Returns:

a corresponding list of StagedFileWrapper objects

ingestFile

public IngestResponse ingestFile(StagedFileWrapper file,
                                 UUID requestId)
                          throws URISyntaxException,
                                 IOException,
                                 IngestResponseException,
                                 BackOffException

ingestFile - ingest a single file

Parameters:

file - - a wrapper around a filename and size

requestId - - a requestId that we'll use to label - if null, we generate one for the user

Returns:

an insert response from the server

Throws:

URISyntaxException - - if the provided account name was illegal and caused a URI construction failure

IOException - - if we have some other network failure

IngestResponseException - - if snowflake encountered error during ingest

BackOffException - - if we have a 503 response

ingestFile

public IngestResponse ingestFile(StagedFileWrapper file,
                                 UUID requestId,
                                 boolean showSkippedFiles)
                          throws URISyntaxException,
                                 IOException,
                                 IngestResponseException,
                                 BackOffException

ingestFile - ingest a single file

Parameters:

file - - a wrapper around a filename and size

requestId - - a requestId that we'll use to label - if null, we generate one for the user

showSkippedFiles - - a flag which returns the files that were skipped when set to true.

Returns:

an insert response from the server

Throws:

URISyntaxException - - if the provided account name was illegal and caused a URI construction failure

IOException - - if we have some other network failure

IngestResponseException - - if snowflake encountered error during ingest

BackOffException - - if we have a 503 response

ingestFiles

public IngestResponse ingestFiles(List<StagedFileWrapper> files,
                                  UUID requestId)
                           throws URISyntaxException,
                                  IOException,
                                  IngestResponseException,
                                  BackOffException

ingestFiles - synchronously sends a request to the ingest service to enqueue these files

Parameters:

files - - list of wrappers around filenames and sizes

requestId - - a requestId that we'll use to label - if null, we generate one for the user

Returns:

an insert response from the server

Throws:

URISyntaxException - - if the provided account name was illegal and caused a URI construction failure

IOException - - if we have some other network failure

IngestResponseException - - if snowflake encountered error during ingest

BackOffException - - if we have a 503 response

ingestFiles

public IngestResponse ingestFiles(List<StagedFileWrapper> files,
                                  UUID requestId,
                                  boolean showSkippedFiles)
                           throws URISyntaxException,
                                  IOException,
                                  IngestResponseException,
                                  BackOffException

ingestFiles - synchronously sends a request to the ingest service to enqueue these files

Parameters:

files - - list of wrappers around filenames and sizes

requestId - - a requestId that we'll use to label - if null, we generate one for the user

showSkippedFiles - - a flag which returns the files that were skipped when set to true.

Returns:

an insert response from the server

Throws:

URISyntaxException - - if the provided account name was illegal and caused a URI construction failure

IOException - - if we have some other network failure

IngestResponseException - - if snowflake encountered error during ingest

BackOffException - - if we have a 503 response

ingestFiles

public IngestResponse ingestFiles(List<StagedFileWrapper> files,
                                  UUID requestId,
                                  boolean showSkippedFiles,
                                  InsertFilesClientInfo clientInfo)
                           throws URISyntaxException,
                                  IOException,
                                  IngestResponseException,
                                  BackOffException

ingestFiles With Client Info - synchronously sends a request to the ingest service to enqueue these files along with clientSequencer and offSetToken.

OffsetToken will be atomically persisted on server(Snowflake) side along with files if the clientSequencer added in this request matches with what Snowflake currently has.

If clientSequencers doesnt match, 400 response code is sent back and no files will be added.

Parameters:

files - - list of wrappers around filenames and sizes

requestId - - a requestId that we'll use to label - if null, we generate one for the user

showSkippedFiles - - a flag which returns the files that were skipped when set to true.

clientInfo - - clientSequencer and offsetToken to pass along with files. Can be null.

Returns:

an insert response from the server

Throws:

URISyntaxException - - if the provided account name was illegal and caused a URI construction failure

IOException - - if we have some other network failure

IngestResponseException - - if snowflake encountered error during ingest

BackOffException - - if we have a 503 response

getHistory

public HistoryResponse getHistory(UUID requestId,
                                  Integer recentSeconds,
                                  String beginMark)
                           throws URISyntaxException,
                                  IOException,
                                  IngestResponseException,
                                  BackOffException

Pings the service to see the current ingest history for this table

Parameters:

requestId - a UUID we use to label the request, if null, one is generated for the user

recentSeconds - history only for items in the recentSeconds window

beginMark - mark from which history should be fetched

Returns:

a response showing the available ingest history from the service

Throws:

URISyntaxException - - if the provided account name was illegal and caused a URI construction failure

IOException - - if we have some other network failure

IngestResponseException - - if snowflake encountered error during ingest

BackOffException - - if we have a 503 response

getHistoryRange

public HistoryRangeResponse getHistoryRange(UUID requestId,
                                            String startTimeInclusive,
                                            String endTimeExclusive)
                                     throws URISyntaxException,
                                            IOException,
                                            IngestResponseException,
                                            BackOffException

Pings the service to see the current ingest history for this table

Parameters:

requestId - a UUID we use to label the request, if null, one is generated for the user

startTimeInclusive - Start time inclusive of scan range, in ISO-8601 format. Missing millisecond part in string will lead to a zero milliseconds. This is a required query parameter, and a 400 will be returned if this query parameter is missing

endTimeExclusive - End time exclusive of scan range. If this query parameter is missing or user provided value is later than current millis, then current millis is used.

Returns:

a response showing the available ingest history from the service

Throws:

URISyntaxException - if the provided account name was illegal and caused a URI construction failure

IOException - - if we have some other network failure

IngestResponseException - - if snowflake encountered error during ingest

BackOffException - if we have a 503 response

configureClient

public ConfigureClientResponse configureClient(UUID requestId)
                                        throws URISyntaxException,
                                               IOException,
                                               IngestResponseException,
                                               BackOffException

Register a snowpipe client and returns the client sequencer

Parameters:

requestId - a UUID we use to label the request, if null, one is generated for the user

Returns:

Throws:

URISyntaxException - - if the provided account name was illegal and caused a URI construction failure

IOException - - if we have some other network failure

IngestResponseException - - if snowflake encountered error during ingest

BackOffException - - if we have a 503 response

getClientStatus

public ClientStatusResponse getClientStatus(UUID requestId)
                                     throws URISyntaxException,
                                            IOException,
                                            IngestResponseException,
                                            BackOffException

Get client status for snowpipe which contains offset token and client sequencer

Parameters:

requestId - a UUID we use to label the request, if null, one is generated for the user

Returns:

Throws:

URISyntaxException - - if the provided account name was illegal and caused a URI construction failure

IOException - - if we have some other network failure

IngestResponseException - - if snowflake encountered error during ingest

BackOffException - - if we have a 503 response

close

public void close()

Closes the resources associated with this object. Resources cannot be reopened, initialize new instance of this class SimpleIngestManager to reopen and start ingesting/monitoring new data.

Specified by:

close in interface AutoCloseable

getRequestBuilder

public RequestBuilder getRequestBuilder()