net.snowflake.ingest.connection

Class RequestBuilder

java.lang.Object

net.snowflake.ingest.connection.RequestBuilder

public class RequestBuilder
extends Object

This class handles constructing the URIs for our requests as well as putting together the payloads we'll be sending

Author:

obabarinsa

Field Summary

Fields

Modifier and Type	Field and Description
static String	CLIENT_NAME
static String	DEFAULT_HOST_SUFFIX
static String	DEFAULT_VERSION
static String	HTTP_HEADER_CONTENT_TYPE_JSON
static String	JAVA_USER_AGENT
static String	JWT_TOKEN_TYPE
static String	OS_INFO_USER_AGENT_FORMAT
static String	SF_HEADER_AUTHORIZATION_TOKEN_TYPE

Constructor Summary

Constructors

Constructor and Description
RequestBuilder(SnowflakeURL url, String userName, KeyPair keyPair) RequestBuilder - constructor used by streaming ingest
RequestBuilder(String accountName, String userName, KeyPair keyPair) RequestBuilder - general usage constructor
RequestBuilder(String accountName, String userName, KeyPair keyPair, String schemeName, String hostName, int portNum) Constructor to use if not intended to use userAgentSuffix.
RequestBuilder(String accountName, String userName, KeyPair keyPair, String schemeName, String hostName, int portNum, String userAgentSuffix) RequestBuilder - this constructor is for testing purposes only
RequestBuilder(String accountName, String userName, String hostName, KeyPair keyPair, String userAgentSuffix) RequestBuilder constructor which uses default schemes, host and port.

Method Summary

Modifier and Type	Method and Description
void	closeResources() Closes the resources being used by RequestBuilder object.
org.apache.http.client.methods.HttpPost	generateConfigureClientRequest(UUID requestID, String pipe) Given a requestId and a pipe, make a configure client request
org.apache.http.client.methods.HttpGet	generateGetClientStatusRequest(UUID requestID, String pipe) Given a requestId and a pipe, make a get client status request
org.apache.http.client.methods.HttpGet	generateHistoryRangeRequest(UUID requestId, String pipe, String startTimeInclusive, String endTimeExclusive) generateHistoryRangeRequest - given a requestId and a pipe, get history for all ingests between time ranges start-end
org.apache.http.client.methods.HttpGet	generateHistoryRequest(UUID requestId, String pipe, Integer recentSeconds, String beginMark) generateHistoryRequest - given a requestId and a pipe, make a history request
org.apache.http.client.methods.HttpPost	generateInsertRequest(UUID requestId, String pipe, List<StagedFileWrapper> files, boolean showSkippedFiles) generateInsertRequest - given a table, stage and list of files, make a request for the insert endpoint
org.apache.http.client.methods.HttpPost	generateInsertRequest(UUID requestId, String pipe, List<StagedFileWrapper> files, boolean showSkippedFiles, Optional<InsertFilesClientInfo> clientInfo) generateInsertRequest - given a pipe, list of files and clientInfo, make a request for the insert endpoint
org.apache.http.client.methods.HttpPost	generateStreamingIngestPostRequest(Map<Object,Object> payload, String endPoint, String message) Generate post request for streaming ingest related APIs
org.apache.http.client.methods.HttpPost	generateStreamingIngestPostRequest(String payload, String endPoint, String message) Generate post request for streaming ingest related APIs

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DEFAULT_HOST_SUFFIX

public static final String DEFAULT_HOST_SUFFIX

See Also:

Constant Field Values

CLIENT_NAME

public static final String CLIENT_NAME

See Also:

Constant Field Values

DEFAULT_VERSION

public static final String DEFAULT_VERSION

See Also:

Constant Field Values

JAVA_USER_AGENT

public static final String JAVA_USER_AGENT

See Also:

Constant Field Values

OS_INFO_USER_AGENT_FORMAT

public static final String OS_INFO_USER_AGENT_FORMAT

See Also:

Constant Field Values

SF_HEADER_AUTHORIZATION_TOKEN_TYPE

public static final String SF_HEADER_AUTHORIZATION_TOKEN_TYPE

See Also:

Constant Field Values

JWT_TOKEN_TYPE

public static final String JWT_TOKEN_TYPE

See Also:

Constant Field Values

HTTP_HEADER_CONTENT_TYPE_JSON

public static final String HTTP_HEADER_CONTENT_TYPE_JSON

See Also:

Constant Field Values

Constructor Detail

RequestBuilder

public RequestBuilder(String accountName,
                      String userName,
                      KeyPair keyPair)

RequestBuilder - general usage constructor

Parameters:

accountName - - the name of the Snowflake account to which we're connecting

userName - - the username of the entity loading files

keyPair - - the Public/Private key pair we'll use to authenticate

RequestBuilder

public RequestBuilder(String accountName,
                      String userName,
                      String hostName,
                      KeyPair keyPair,
                      String userAgentSuffix)

RequestBuilder constructor which uses default schemes, host and port.

Parameters:

accountName - - the name of the Snowflake account to which we're connecting

userName - - the username of the entity loading files

keyPair - - the Public/Private key pair we'll use to authenticate

RequestBuilder

public RequestBuilder(String accountName,
                      String userName,
                      KeyPair keyPair,
                      String schemeName,
                      String hostName,
                      int portNum)

Constructor to use if not intended to use userAgentSuffix. i.e. User-Agent HTTP header suffix part is null, (The default one is still valid, check out #defaultUserAgent)

Parameters:

accountName - - the account name to which we're connecting

userName - - for whom are we connecting?

keyPair - - our auth credentials

schemeName - - are we HTTP or HTTPS?

hostName - - the host for this snowflake instance

portNum - - the port number

RequestBuilder

public RequestBuilder(String accountName,
                      String userName,
                      KeyPair keyPair,
                      String schemeName,
                      String hostName,
                      int portNum,
                      String userAgentSuffix)

RequestBuilder - this constructor is for testing purposes only

Parameters:

accountName - - the account name to which we're connecting

userName - - for whom are we connecting?

keyPair - - our auth credentials

schemeName - - are we HTTP or HTTPS?

hostName - - the host for this snowflake instance

portNum - - the port number

userAgentSuffix - - The suffix part of HTTP Header User-Agent

RequestBuilder

public RequestBuilder(SnowflakeURL url,
                      String userName,
                      KeyPair keyPair)

RequestBuilder - constructor used by streaming ingest

Parameters:

url - - the Snowflake account to which we're connecting

userName - - the username of the entity loading files

keyPair - - the Public/Private key pair we'll use to authenticate

Method Detail

generateInsertRequest

public org.apache.http.client.methods.HttpPost generateInsertRequest(UUID requestId,
                                                                     String pipe,
                                                                     List<StagedFileWrapper> files,
                                                                     boolean showSkippedFiles)
                                                              throws URISyntaxException

generateInsertRequest - given a table, stage and list of files, make a request for the insert endpoint

Parameters:

requestId - a UUID we will use to label this request

pipe - a fully qualified pipe name

files - a list of files

showSkippedFiles - a boolean which returns skipped files when set to true

Returns:

a post request with all the data we need

Throws:

URISyntaxException - if the URI components provided are improper

generateInsertRequest

public org.apache.http.client.methods.HttpPost generateInsertRequest(UUID requestId,
                                                                     String pipe,
                                                                     List<StagedFileWrapper> files,
                                                                     boolean showSkippedFiles,
                                                                     Optional<InsertFilesClientInfo> clientInfo)
                                                              throws URISyntaxException

generateInsertRequest - given a pipe, list of files and clientInfo, make a request for the insert endpoint

Parameters:

requestId - a UUID we will use to label this request

pipe - a fully qualified pipe name

files - a list of files

showSkippedFiles - a boolean which returns skipped files when set to true

clientInfo -

Returns:

a post request with all the data we need

Throws:

URISyntaxException - if the URI components provided are improper

generateHistoryRequest

public org.apache.http.client.methods.HttpGet generateHistoryRequest(UUID requestId,
                                                                     String pipe,
                                                                     Integer recentSeconds,
                                                                     String beginMark)
                                                              throws URISyntaxException

generateHistoryRequest - given a requestId and a pipe, make a history request

Parameters:

requestId - a UUID we will use to label this request

pipe - a fully qualified pipe name

recentSeconds - history only for items in the recentSeconds window

beginMark - mark from which history should be fetched

Throws:

URISyntaxException - - If the URI components provided are improper

generateHistoryRangeRequest

public org.apache.http.client.methods.HttpGet generateHistoryRangeRequest(UUID requestId,
                                                                          String pipe,
                                                                          String startTimeInclusive,
                                                                          String endTimeExclusive)
                                                                   throws URISyntaxException

generateHistoryRangeRequest - given a requestId and a pipe, get history for all ingests between time ranges start-end

Parameters:

requestId - a UUID we will use to label this request

pipe - a fully qualified pipe name

startTimeInclusive - Start time inclusive of scan range, in ISO-8601 format. Missing millisecond part in string will lead to a zero milliseconds. This is a required query parameter, and a 400 will be returned if this query parameter is missing

endTimeExclusive - End time exclusive of scan range. If this query parameter is missing or user provided value is later than current millis, then current millis is used.

Returns:

URI for the insert request

Throws:

URISyntaxException

generateStreamingIngestPostRequest

public org.apache.http.client.methods.HttpPost generateStreamingIngestPostRequest(String payload,
                                                                                  String endPoint,
                                                                                  String message)

Generate post request for streaming ingest related APIs

Parameters:

payload - POST request payload as string

endPoint - REST API endpoint

message - error message if there are failures during HTTP building

Returns:

URI for the POST request

generateConfigureClientRequest

public org.apache.http.client.methods.HttpPost generateConfigureClientRequest(UUID requestID,
                                                                              String pipe)
                                                                       throws URISyntaxException

Given a requestId and a pipe, make a configure client request

Parameters:

requestID - a UUID we will use to label this request

pipe - a fully qualified pipe name

Returns:

configure client request

Throws:

URISyntaxException

generateStreamingIngestPostRequest

public org.apache.http.client.methods.HttpPost generateStreamingIngestPostRequest(Map<Object,Object> payload,
                                                                                  String endPoint,
                                                                                  String message)

Generate post request for streaming ingest related APIs

Parameters:

payload - POST request payload

endPoint - REST API endpoint

message - error message if there are failures during HTTP building

Returns:

URI for the POST request

generateGetClientStatusRequest

public org.apache.http.client.methods.HttpGet generateGetClientStatusRequest(UUID requestID,
                                                                             String pipe)
                                                                      throws URISyntaxException

Given a requestId and a pipe, make a get client status request

Parameters:

requestID - UUID

pipe - a fully qualified pipe name

Returns:

get client status request

Throws:

URISyntaxException

closeResources

public void closeResources()

Closes the resources being used by RequestBuilder object. SecurityManager is one such resource which uses a threadpool which needs to be shutdown once SimpleIngestManager is done interacting with Snowpipe Service (Rest APIs)

Throws:

Exception