net.snowflake.ingest

Class SimpleIngestManager

java.lang.Object

net.snowflake.ingest.SimpleIngestManager

public class SimpleIngestManager
extends Object

This class provides a basic, low-level abstraction over the Snowflake Ingest Service REST api

Usage of this class delegates all exception and state handling to the developer

Author:

obabarinsa

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	SimpleIngestManager.Builder This Builder allows someone to configure a SimpleIngestManager prior to instantiating the manager

Constructor Summary

Constructors

Constructor and Description
SimpleIngestManager(String account, String user, String pipe, KeyPair keyPair) Deprecated.
SimpleIngestManager(String account, String user, String pipe, KeyPair keyPair, String schemeName, String hostName, int port) Deprecated.
SimpleIngestManager(String account, String user, String pipe, PrivateKey privateKey) Constructs a SimpleIngestManager for a given user in a specific account In addition, this also takes takes the target table and source stage Finally, it also requires a valid private key registered with Snowflake DB Note: this method only takes in account parameter and derive the hostname, i.e.
SimpleIngestManager(String account, String user, String pipe, PrivateKey privateKey, String schemeName, String hostName, int port) Constructs a SimpleIngestManager for a given user in a specific account In addition, this also takes takes the target table and source stage Finally, it also requires a valid private key registered with Snowflake DB

Method Summary

Modifier and Type	Method and Description
String	getAccount() getAccount - Gives back the name of the account that this IngestManager is targeting
HistoryResponse	getHistory(UUID requestId) Pings the service to see the current ingest history for this table
HistoryResponse	getHistory(UUID requestId, Integer recentSeconds, String beginMark) Pings the service to see the current ingest history for this table
HistoryRangeResponse	getHistoryRange(UUID requestId, String startTimeInclusive, String endTimeExclusive) Pings the service to see the current ingest history for this table
String	getPipe() getPipe - gives back the pipe which we are using
String	getUser() getUser - gives back the user on behalf of which this ingest manager is loading
IngestResponse	ingestFile(StagedFileWrapper file, UUID requestId) ingestFile - ingest a single file
IngestResponse	ingestFiles(List<StagedFileWrapper> files, UUID requestId) ingestFiles - synchronously sends a request to the ingest service to enqueue these files
static List<StagedFileWrapper>	wrapFilepaths(Set<String> filenames) wrapFilepaths - convenience method to take a list of filenames and produce a list of FileWrappers with unset size

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SimpleIngestManager

@Deprecated
public SimpleIngestManager(String account,
                                       String user,
                                       String pipe,
                                       KeyPair keyPair)

Deprecated.

Constructs a SimpleIngestManager for a given user in a specific account In addition, this also takes takes the target table and source stage Finally, it also requires a valid KeyPair object registered with Snowflake DB This method is deprecated, please use the constructor that only requires PrivateKey instead of KeyPair.

Parameters:

account - The account into which we're loading Note: account should not include region or cloud provider info. e.g. if host is testaccount.us-east-1.azure .snowflakecomputing.com, account should be testaccount

user - the user performing this load

pipe - the fully qualified name of the pipe

keyPair - the KeyPair we'll use to sign JWT tokens

SimpleIngestManager

public SimpleIngestManager(String account,
                           String user,
                           String pipe,
                           PrivateKey privateKey)
                    throws InvalidKeySpecException,
                           NoSuchAlgorithmException

Constructs a SimpleIngestManager for a given user in a specific account In addition, this also takes takes the target table and source stage Finally, it also requires a valid private key registered with Snowflake DB Note: this method only takes in account parameter and derive the hostname, i.e. testaccount.snowfakecomputing.com. If your deployment is not aws us-west, please use the constructor that accept hostname as argument

Parameters:

account - The account into which we're loading Note: account should not include region or cloud provider info. e.g. if host is testaccount.us-east-1.azure .snowflakecomputing.com, account should be testaccount. If this is the case, you should use the constructor that accepts hostname as argument

user - the user performing this load

pipe - the fully qualified name of the pipe

privateKey - the private key we'll use to sign JWT tokens

Throws:

NoSuchAlgorithmException - if can't create key factory by using RSA algorithm

InvalidKeySpecException - if private key or public key is invalid

SimpleIngestManager

@Deprecated
public SimpleIngestManager(String account,
                                       String user,
                                       String pipe,
                                       KeyPair keyPair,
                                       String schemeName,
                                       String hostName,
                                       int port)

Deprecated.

Constructs a SimpleIngestManager for a given user in a specific account In addition, this also takes takes the target table and source stage Finally, it also requires a valid KeyPair object registered with Snowflake DB This method is deprecated, please use the constructor that only requires PrivateKey instead of KeyPair.

Parameters:

account - the account into which we're loading Note: account should not include region or cloud provider info. e.g. if host is testaccount.us-east-1.azure .snowflakecomputing.com account should be testaccount If this is the case, you should use the constructor that accepts hostname as argument

user - the user performing this load

pipe - the fully qualified name of the pipe

keyPair - the KeyPair we'll use to sign JWT tokens

schemeName - http or https

hostName - the hostname

port - the port number

SimpleIngestManager

public SimpleIngestManager(String account,
                           String user,
                           String pipe,
                           PrivateKey privateKey,
                           String schemeName,
                           String hostName,
                           int port)
                    throws NoSuchAlgorithmException,
                           InvalidKeySpecException

Constructs a SimpleIngestManager for a given user in a specific account In addition, this also takes takes the target table and source stage Finally, it also requires a valid private key registered with Snowflake DB

Parameters:

account - the account into which we're loading Note: account should not include region or cloud provider info. e.g. if host is testaccount.us-east-1.azure .snowflakecomputing.com, account should be testaccount

user - the user performing this load

pipe - the fully qualified name of the pipe

privateKey - the private key we'll use to sign JWT tokens

schemeName - http or https

hostName - the hostname i.e. testaccount.us-east-1.azure .snowflakecomputing.com

port - the port number

Throws:

NoSuchAlgorithmException - if can't create key factory by using RSA algorithm

InvalidKeySpecException - if private key or public key is invalid

Method Detail

getAccount

public String getAccount()

getAccount - Gives back the name of the account that this IngestManager is targeting

Returns:

the name of the account

getUser

public String getUser()

getUser - gives back the user on behalf of which this ingest manager is loading

Returns:

the user name

getPipe

public String getPipe()

getPipe - gives back the pipe which we are using

Returns:

the pipe name

wrapFilepaths

public static List<StagedFileWrapper> wrapFilepaths(Set<String> filenames)

wrapFilepaths - convenience method to take a list of filenames and produce a list of FileWrappers with unset size

Parameters:

filenames - the filenames you want to wrap up

Returns:

a corresponding list of StagedFileWrapper objects

ingestFile

public IngestResponse ingestFile(StagedFileWrapper file,
                                 UUID requestId)
                          throws URISyntaxException,
                                 IOException,
                                 Exception

ingestFile - ingest a single file

Parameters:

file - - a wrapper around a filename and size

requestId - - a requestId that we'll use to label - if null, we generate one for the user

Returns:

an insert response from the server

Throws:

BackOffException - - if we have a 503 response

IOException - - if we have some other network failure

URISyntaxException - - if the provided account name was illegal and caused a URI construction failure

Exception

ingestFiles

public IngestResponse ingestFiles(List<StagedFileWrapper> files,
                                  UUID requestId)
                           throws URISyntaxException,
                                  IOException,
                                  IngestResponseException

ingestFiles - synchronously sends a request to the ingest service to enqueue these files

Parameters:

files - - list of wrappers around filenames and sizes

requestId - - a requestId that we'll use to label - if null, we generate one for the user

Returns:

an insert response from the server

Throws:

BackOffException - - if we have a 503 response

IOException - - if we have some other network failure

IngestResponseException - - if snowflake encountered error during ingest

URISyntaxException - - if the provided account name was illegal and caused a URI construction failure

getHistory

public HistoryResponse getHistory(UUID requestId,
                                  Integer recentSeconds,
                                  String beginMark)
                           throws URISyntaxException,
                                  IOException,
                                  IngestResponseException

Pings the service to see the current ingest history for this table

Parameters:

requestId - a UUID we use to label the request, if null, one is generated for the user

recentSeconds - history only for items in the recentSeconds window

beginMark - mark from which history should be fetched

Returns:

a response showing the available ingest history from the service

Throws:

BackOffException - - if we have a 503 response

IOException - - if we have some other network failure

IngestResponseException - - if snowflake encountered a service error

URISyntaxException - - if the provided account name was illegal and caused a URI construction failure

getHistory

public HistoryResponse getHistory(UUID requestId)
                           throws URISyntaxException,
                                  IOException,
                                  IngestResponseException

Pings the service to see the current ingest history for this table

Parameters:

requestId - a UUID we use to label the request, if null, one is generated for the user

Returns:

a response showing the available ingest history from the service

Throws:

BackOffException - - if we have a 503 response

IOException - - if we have some other network failure

IngestResponseException - -if snowflake encountered a service error

URISyntaxException - - if the provided account name was illegal and caused a URI construction failure

getHistoryRange

public HistoryRangeResponse getHistoryRange(UUID requestId,
                                            String startTimeInclusive,
                                            String endTimeExclusive)
                                     throws URISyntaxException,
                                            IOException,
                                            IngestResponseException

Pings the service to see the current ingest history for this table

Parameters:

requestId - a UUID we use to label the request, if null, one is generated for the user

startTimeInclusive - Start time inclusive of scan range, in ISO-8601 format. Missing millisecond part in string will lead to a zero milliseconds. This is a required query parameter, and a 400 will be returned if this query parameter is missing

endTimeExclusive - End time exclusive of scan range. If this query parameter is missing or user provided value is later than current millis, then current millis is used.

Returns:

a response showing the available ingest history from the service

Throws:

BackOffException - - if we have a 503 response

IOException - - if we have some other network failure

IngestResponseException - -if snowflake encountered a service error

URISyntaxException - - if the provided account name was illegal and caused a URI construction failure