Package net.snowflake.client.jdbc.cloud.storage

Interface SnowflakeStorageClient

    • Method Detail

      • getMaxRetries

        int getMaxRetries()
        Returns:
        Returns the Max number of retry attempts

      • getRetryBackoffMaxExponent

        int getRetryBackoffMaxExponent()
        Returns the max exponent for multiplying backoff with the power of 2, the value of 4 will give us 16secs as the max number of time to sleep before retry
        Returns:
        Returns the exponent

      • getRetryBackoffMin

        int getRetryBackoffMin()
        Returns:
        Returns the min number of milliseconds to sleep before retry

      • isEncrypting

        boolean isEncrypting()
        Returns:
        Returns true if encryption is enabled

      • getEncryptionKeySize

        int getEncryptionKeySize()
        Returns:
        Returns the size of the encryption key

      • requirePresignedUrl

        default boolean requirePresignedUrl()
        Returns:
        Whether this client requires the use of presigned URLs for upload and download instead of credentials that work for all files uploaded/ downloaded to a stage path. True for GCS.

      • renew

        void renew​(Map<?,​?> stageCredentials)
    throws SnowflakeSQLException
        Re-creates the encapsulated storage client with a fresh access token
        Parameters:
        stageCredentials - a Map (as returned by GS) which contains the new credential properties
        Throws:
        SnowflakeSQLException - failure to renew the storage client

      • shutdown

        void shutdown()
        shuts down the client

      • listObjects

        StorageObjectSummaryCollection listObjects​(String remoteStorageLocation,
                                           String prefix)
                                    throws StorageProviderException
        For a set of remote storage objects under a remote location and a given prefix/path returns their properties wrapped in ObjectSummary objects
        Parameters:
        remoteStorageLocation - location, i.e. bucket for S3
        prefix - the prefix to list
        Returns:
        a collection of storage summary objects
        Throws:
        StorageProviderException - cloud storage provider error

      • getObjectMetadata

        StorageObjectMetadata getObjectMetadata​(String remoteStorageLocation,
                                        String prefix)
                                 throws StorageProviderException
        Returns the metadata properties for a remote storage object
        Parameters:
        remoteStorageLocation - location, i.e. bucket for S3
        prefix - the prefix/path of the object to retrieve
        Returns:
        storage metadata object
        Throws:
        StorageProviderException - cloud storage provider error

      • download

        void download​(SFSession connection,
              String command,
              String localLocation,
              String destFileName,
              int parallelism,
              String remoteStorageLocation,
              String stageFilePath,
              String stageRegion,
              String presignedUrl,
              String queryId)
       throws SnowflakeSQLException
        Download a file from remote storage.
        Parameters:
        connection - connection object
        command - command to download file
        localLocation - local file path
        destFileName - destination file name
        parallelism - number of threads for parallel downloading
        remoteStorageLocation - remote storage location, i.e. bucket for S3
        stageFilePath - stage file path
        stageRegion - region name where the stage persists
        presignedUrl - presigned URL for download. Used by GCP.
        queryId - last query id
        Throws:
        SnowflakeSQLException - download failure

      • downloadToStream

        InputStream downloadToStream​(SFSession connection,
                             String command,
                             int parallelism,
                             String remoteStorageLocation,
                             String stageFilePath,
                             String stageRegion,
                             String presignedUrl,
                             String queryId)
                      throws SnowflakeSQLException
        Download a file from remote storage
        Parameters:
        connection - connection object
        command - command to download file
        parallelism - number of threads for parallel downloading
        remoteStorageLocation - remote storage location, i.e. bucket for s3
        stageFilePath - stage file path
        stageRegion - region name where the stage persists
        presignedUrl - presigned URL for download. Used by GCP.
        queryId - last query id
        Returns:
        input file stream
        Throws:
        SnowflakeSQLException - when download failure

      • upload

        void upload​(SFSession connection,
            String command,
            int parallelism,
            boolean uploadFromStream,
            String remoteStorageLocation,
            File srcFile,
            String destFileName,
            InputStream inputStream,
            FileBackedOutputStream fileBackedOutputStream,
            StorageObjectMetadata meta,
            String stageRegion,
            String presignedUrl,
            String queryId)
     throws SnowflakeSQLException
        Upload a file (-stream) to remote storage
        Parameters:
        connection - connection object
        command - upload command
        parallelism - number of threads do parallel uploading
        uploadFromStream - true if upload source is stream
        remoteStorageLocation - s3 bucket name
        srcFile - source file if not uploading from a stream
        destFileName - file name on remote storage after upload
        inputStream - stream used for uploading if fileBackedOutputStream is null
        fileBackedOutputStream - stream used for uploading if not null
        meta - object meta data
        stageRegion - region name where the stage persists
        presignedUrl - presigned URL for upload. Used by GCP.
        queryId - last query id
        Throws:
        SnowflakeSQLException - if upload failed even after retry

      • uploadWithPresignedUrlWithoutConnection

        default void uploadWithPresignedUrlWithoutConnection​(int networkTimeoutInMilli,
                                                     HttpClientSettingsKey ocspModeAndProxyKey,
                                                     int parallelism,
                                                     boolean uploadFromStream,
                                                     String remoteStorageLocation,
                                                     File srcFile,
                                                     String destFileName,
                                                     InputStream inputStream,
                                                     FileBackedOutputStream fileBackedOutputStream,
                                                     StorageObjectMetadata meta,
                                                     String stageRegion,
                                                     String presignedUrl,
                                                     String queryId)
                                              throws SnowflakeSQLException
        Upload a file (-stream) to remote storage with Pre-signed URL without JDBC connection.

        NOTE: This function is only supported when pre-signed URL is used.

        Parameters:
        networkTimeoutInMilli - Network timeout for the upload
        ocspModeAndProxyKey - OCSP mode and proxy settings for the upload.
        parallelism - number of threads do parallel uploading
        uploadFromStream - true if upload source is stream
        remoteStorageLocation - s3 bucket name
        srcFile - source file if not uploading from a stream
        destFileName - file name on remote storage after upload
        inputStream - stream used for uploading if fileBackedOutputStream is null
        fileBackedOutputStream - stream used for uploading if not null
        meta - object meta data
        stageRegion - region name where the stage persists
        presignedUrl - presigned URL for upload. Used by GCP.
        queryId - last query id
        Throws:
        SnowflakeSQLException - if upload failed even after retry

      • handleStorageException

        @Deprecated
default void handleStorageException​(Exception ex,
                                    int retryCount,
                                    String operation,
                                    SFSession connection,
                                    String command)
                             throws SnowflakeSQLException
        Deprecated.
        use handleStorageException(Exception, int, String, SFSession, String, String)
        Handles exceptions thrown by the remote storage provider
        Parameters:
        ex - the exception to handle
        retryCount - current number of retries, incremented by the caller before each call
        operation - string that indicates the function/operation that was taking place, when the exception was raised, for example "upload"
        connection - the current SFSession object used by the client
        command - the command attempted at the time of the exception
        Throws:
        SnowflakeSQLException - exceptions that were not handled, or retried past what the retry policy allows, are propagated

      • handleStorageException

        void handleStorageException​(Exception ex,
                            int retryCount,
                            String operation,
                            SFSession connection,
                            String command,
                            String queryId)
                     throws SnowflakeSQLException
        Handles exceptions thrown by the remote storage provider
        Parameters:
        ex - the exception to handle
        retryCount - current number of retries, incremented by the caller before each call
        operation - string that indicates the function/operation that was taking place, when the exception was raised, for example "upload"
        connection - the current SFSession object used by the client
        command - the command attempted at the time of the exception
        queryId - last query id
        Throws:
        SnowflakeSQLException - exceptions that were not handled, or retried past what the retry policy allows, are propagated

      • getMatdescKey

        String getMatdescKey()
        Returns the material descriptor key
        Returns:
        the material descriptor key

      • addEncryptionMetadata

        void addEncryptionMetadata​(StorageObjectMetadata meta,
                           MatDesc matDesc,
                           byte[] ivData,
                           byte[] encryptedKey,
                           long contentLength)
        Adds encryption metadata to the StorageObjectMetadata object for AES-ECB/AES-CBC
        Parameters:
        meta - the storage metadata object to add the encryption info to
        matDesc - the material descriptor
        ivData - the initialization vector
        encryptedKey - the encrypted content encryption key
        contentLength - the length of the encrypted content

      • addEncryptionMetadataForGcm

        @SnowflakeJdbcInternalApi
default void addEncryptionMetadataForGcm​(StorageObjectMetadata meta,
                                         MatDesc matDesc,
                                         byte[] encryptedKey,
                                         byte[] dataIvBytes,
                                         byte[] keyIvBytes,
                                         byte[] keyAad,
                                         byte[] dataAad,
                                         long contentLength)
        Adds encryption metadata to the StorageObjectMetadata object for AES-GCM/AES-GCM
        Parameters:
        meta - the storage metadata object to add the encryption info to
        matDesc - the material descriptor
        encryptedKey - encrypted key
        dataIvBytes - the initialization vector for data
        keyIvBytes - the initialization vector for file key
        keyAad - the additional authenticated data for file key
        dataAad - the additional authenticated data for data
        contentLength - the length of the encrypted content

      • addDigestMetadata

        void addDigestMetadata​(StorageObjectMetadata meta,
                       String digest)
        Adds digest metadata to the StorageObjectMetadata object
        Parameters:
        meta - the storage metadata object to add the digest to
        digest - the digest metadata to add

      • getDigestMetadata

        String getDigestMetadata​(StorageObjectMetadata meta)
        Gets digest metadata to the StorageObjectMetadata object
        Parameters:
        meta - the metadata object to extract the digest metadata from
        Returns:
        the digest metadata value

      • addStreamingIngestMetadata

        void addStreamingIngestMetadata​(StorageObjectMetadata meta,
                                String clientName,
                                String clientKey)
        Adds streaming ingest metadata to the StorageObjectMetadata object, used for streaming ingest per client billing calculation
        Parameters:
        meta - the storage metadata object to add the digest to
        clientName - streaming ingest client name
        clientKey - streaming ingest client key, provided by Snowflake

      • getStreamingIngestClientName

        String getStreamingIngestClientName​(StorageObjectMetadata meta)
        Gets streaming ingest client name to the StorageObjectMetadata object

      • getStreamingIngestClientKey

        String getStreamingIngestClientKey​(StorageObjectMetadata meta)
        Gets streaming ingest client key to the StorageObjectMetadata object