Package com.microsoft.azure.kusto.ingest

Class ManagedStreamingIngestClient

java.lang.Object

com.microsoft.azure.kusto.ingest.ManagedStreamingIngestClient

All Implemented Interfaces:

IngestClient, Closeable, AutoCloseable

public class ManagedStreamingIngestClient
extends Object
implements IngestClient

ManagedStreamingIngestClient

This class combines a managed streaming client with a queued streaming client, to create an optimized experience. Since the streaming client communicates directly with the engine, it's more prone to failure, so this class holds both a streaming client and a queued client. It tries 3 times using the streaming client, after which it falls back to the queued streaming client in case of failure. If the size of the stream is bigger than 4194304, it will fall back to the queued streaming client.

Note that ingestFromBlob behaves differently from the other methods - since a blob already exists it makes more sense to enqueue it rather than downloading and streaming it, thus ManagedStreamingIngestClient skips the streaming retries and sends it directly to the queued client.

Field Summary

Fields

Modifier and Type	Field	Description
static int	ATTEMPT_COUNT
static int	MAX_STREAMING_SIZE_BYTES

Constructor Summary

Constructors

Constructor	Description
ManagedStreamingIngestClient(ConnectionStringBuilder ingestionEndpointConnectionStringBuilder, ConnectionStringBuilder queryEndpointConnectionStringBuilder)	Deprecated. - This method is slated to be private.
ManagedStreamingIngestClient(ConnectionStringBuilder ingestionEndpointConnectionStringBuilder, ConnectionStringBuilder queryEndpointConnectionStringBuilder, @Nullable HttpClientProperties properties)	Deprecated. - This method is slated to be private.
ManagedStreamingIngestClient(com.microsoft.azure.kusto.ingest.ResourceManager resourceManager, com.microsoft.azure.kusto.ingest.AzureStorageClient storageClient, StreamingClient streamingClient)	Deprecated. - This method is slated to be private.

Method Summary

Modifier and Type	Method	Description
void	close()
static ManagedStreamingIngestClient	fromDmConnectionString(ConnectionStringBuilder dmConnectionString)	Deprecated. - Ingest clients now automatically deduce the endpoint, use ManagedStreamingIngestClient(ConnectionStringBuilder, HttpClientProperties) instead.
static ManagedStreamingIngestClient	fromDmConnectionString(ConnectionStringBuilder dmConnectionString, @Nullable HttpClientProperties properties)	Deprecated. - Ingest clients now automatically deduce the endpoint, use ManagedStreamingIngestClient(ConnectionStringBuilder, HttpClientProperties) instead.
static ManagedStreamingIngestClient	fromEngineConnectionString(ConnectionStringBuilder engineConnectionString)	Deprecated. - Ingest clients now automatically deduce the endpoint, use ManagedStreamingIngestClient(ConnectionStringBuilder, HttpClientProperties) instead.
static ManagedStreamingIngestClient	fromEngineConnectionString(ConnectionStringBuilder engineConnectionString, @Nullable HttpClientProperties properties)	Deprecated. - Ingest clients now automatically deduce the endpoint, use ManagedStreamingIngestClient(ConnectionStringBuilder, HttpClientProperties) instead.
IngestionResult	ingestFromBlob(BlobSourceInfo blobSourceInfo, IngestionProperties ingestionProperties)	Ingest data from a blob storage into Kusto database.
IngestionResult	ingestFromFile(FileSourceInfo fileSourceInfo, IngestionProperties ingestionProperties)	Ingest data from a file into Kusto database.
IngestionResult	ingestFromResultSet(ResultSetSourceInfo resultSetSourceInfo, IngestionProperties ingestionProperties)	Ingest data from a Result Set into Kusto database.
IngestionResult	ingestFromStream(StreamSourceInfo streamSourceInfo, IngestionProperties ingestionProperties)	Ingest data from an input stream, into Kusto database.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

ATTEMPT_COUNT

public static final int ATTEMPT_COUNT

See Also:

Constant Field Values

MAX_STREAMING_SIZE_BYTES

public static final int MAX_STREAMING_SIZE_BYTES

See Also:

Constant Field Values

Constructor Detail

ManagedStreamingIngestClient

public ManagedStreamingIngestClient(ConnectionStringBuilder ingestionEndpointConnectionStringBuilder,
                                    ConnectionStringBuilder queryEndpointConnectionStringBuilder)
                             throws URISyntaxException

Deprecated.

- This method is slated to be private. Use IngestClientFactory.createManagedStreamingIngestClient(ConnectionStringBuilder, ConnectionStringBuilder) instead.

Throws:

URISyntaxException

ManagedStreamingIngestClient

public ManagedStreamingIngestClient(ConnectionStringBuilder ingestionEndpointConnectionStringBuilder,
                                    ConnectionStringBuilder queryEndpointConnectionStringBuilder,
                                    @Nullable
                                    @Nullable HttpClientProperties properties)
                             throws URISyntaxException

Deprecated.

- This method is slated to be private. Use IngestClientFactory.createManagedStreamingIngestClient(ConnectionStringBuilder, ConnectionStringBuilder, HttpClientProperties) instead. This constructor should only be used for advanced cases. If your endpoints are standard, or you do not know, use ManagedStreamingIngestClient(ConnectionStringBuilder, HttpClientProperties))} instead.

Parameters:

ingestionEndpointConnectionStringBuilder - - Endpoint for ingesting data, usually starts with "https://ingest-"

queryEndpointConnectionStringBuilder - - Endpoint for querying data, does not include "ingest-"

properties - - Additional properties to configure the http client

Throws:

URISyntaxException - if the connection string is invalid

ManagedStreamingIngestClient

public ManagedStreamingIngestClient(com.microsoft.azure.kusto.ingest.ResourceManager resourceManager,
                                    com.microsoft.azure.kusto.ingest.AzureStorageClient storageClient,
                                    StreamingClient streamingClient)

Deprecated.

- This method is slated to be private. Use IngestClientFactory.createManagedStreamingIngestClient(ConnectionStringBuilder) instead.

Method Detail

fromDmConnectionString

public static ManagedStreamingIngestClient fromDmConnectionString(ConnectionStringBuilder dmConnectionString)
                                                           throws URISyntaxException

Deprecated.

- Ingest clients now automatically deduce the endpoint, use ManagedStreamingIngestClient(ConnectionStringBuilder, HttpClientProperties) instead. Creates a new ManagedStreamingIngestClient from a DM connection string, with default http client properties. This method infers the engine connection string from the DM connection string. For advanced usage, use ManagedStreamingIngestClient(ConnectionStringBuilder, ConnectionStringBuilder)

Parameters:

dmConnectionString - dm connection string

Returns:

a new ManagedStreamingIngestClient

Throws:

URISyntaxException - if the connection string is invalid

fromDmConnectionString

public static ManagedStreamingIngestClient fromDmConnectionString(ConnectionStringBuilder dmConnectionString,
                                                                  @Nullable
                                                                  @Nullable HttpClientProperties properties)
                                                           throws URISyntaxException

Deprecated.

- Ingest clients now automatically deduce the endpoint, use ManagedStreamingIngestClient(ConnectionStringBuilder, HttpClientProperties) instead. Creates a new ManagedStreamingIngestClient from a DM connection string. This method infers the engine connection string from the DM connection string. For advanced usage, use ManagedStreamingIngestClient(ConnectionStringBuilder, ConnectionStringBuilder)

Parameters:

dmConnectionString - dm connection string

properties - additional properties to configure the http client

Returns:

a new ManagedStreamingIngestClient

Throws:

URISyntaxException - if the connection string is invalid

fromEngineConnectionString

public static ManagedStreamingIngestClient fromEngineConnectionString(ConnectionStringBuilder engineConnectionString)
                                                               throws URISyntaxException

Deprecated.

- Ingest clients now automatically deduce the endpoint, use ManagedStreamingIngestClient(ConnectionStringBuilder, HttpClientProperties) instead. Creates a new ManagedStreamingIngestClient from an engine connection string, with default http client properties. This method infers the DM connection string from the engine connection string. For advanced usage, use ManagedStreamingIngestClient(ConnectionStringBuilder, ConnectionStringBuilder)

Parameters:

engineConnectionString - engine connection string

Returns:

a new ManagedStreamingIngestClient

Throws:

URISyntaxException - if the connection string is invalid

fromEngineConnectionString

public static ManagedStreamingIngestClient fromEngineConnectionString(ConnectionStringBuilder engineConnectionString,
                                                                      @Nullable
                                                                      @Nullable HttpClientProperties properties)
                                                               throws URISyntaxException

Deprecated.

- Ingest clients now automatically deduce the endpoint, use ManagedStreamingIngestClient(ConnectionStringBuilder, HttpClientProperties) instead. Creates a new ManagedStreamingIngestClient from an engine connection string. This method infers the DM connection string from the engine connection string. For advanced usage, use ManagedStreamingIngestClient(ConnectionStringBuilder, ConnectionStringBuilder)

Parameters:

engineConnectionString - engine connection string

properties - additional properties to configure the http client

Returns:

a new ManagedStreamingIngestClient

Throws:

URISyntaxException - if the connection string is invalid

ingestFromFile

public IngestionResult ingestFromFile(FileSourceInfo fileSourceInfo,
                                      IngestionProperties ingestionProperties)
                               throws IngestionClientException,
                                      IngestionServiceException

Description copied from interface: IngestClient

Ingest data from a file into Kusto database.

This method ingests the data from a given file, described in fileSourceInfo, into Kusto database, according to the properties mentioned in ingestionProperties

Specified by:

ingestFromFile in interface IngestClient

Parameters:

fileSourceInfo - The specific SourceInfo to be ingested

ingestionProperties - Settings used to customize the ingestion operation

Returns:

IngestionResult object including the ingestion result

Throws:

IngestionClientException - An exception originating from a client activity

IngestionServiceException - An exception returned from the service

See Also:

FileSourceInfo, IngestionProperties

ingestFromBlob

public IngestionResult ingestFromBlob(BlobSourceInfo blobSourceInfo,
                                      IngestionProperties ingestionProperties)
                               throws IngestionClientException,
                                      IngestionServiceException

Ingest data from a blob storage into Kusto database.

This method ingests the data from a given blob, described in blobSourceInfo, into Kusto database, according to the properties mentioned in ingestionProperties

This method behaves differently from the rest for ManagedStreamingIngestClient - since a blob already exists it makes more sense to enqueue it rather than downloading and streaming it, thus ManagedStreamingIngestClient skips the streaming retries and sends it directly to the queued client.

Specified by:

ingestFromBlob in interface IngestClient

Parameters:

blobSourceInfo - The specific SourceInfo to be ingested

ingestionProperties - Settings used to customize the ingestion operation

Returns:

IngestionResult object including the ingestion result

Throws:

IngestionClientException - An exception originating from a client activity

IngestionServiceException - An exception returned from the service

See Also:

BlobSourceInfo, IngestionProperties

ingestFromResultSet

public IngestionResult ingestFromResultSet(ResultSetSourceInfo resultSetSourceInfo,
                                           IngestionProperties ingestionProperties)
                                    throws IngestionClientException,
                                           IngestionServiceException

Description copied from interface: IngestClient

Ingest data from a Result Set into Kusto database.

This method ingests the data from a given Result Set, described in resultSetSourceInfo, into Kusto database, according to the properties mentioned in ingestionProperties

Ingesting from ResultSet is equivalent to ingesting from a csv stream. The DataFormat should be empty or set to "csv", and the mapping, should it be provided, should be csv mapping.

Specified by:

ingestFromResultSet in interface IngestClient

Parameters:

resultSetSourceInfo - The specific SourceInfo to be ingested

ingestionProperties - Settings used to customize the ingestion operation

Returns:

IngestionResult object including the ingestion result

Throws:

IngestionClientException - An exception originating from a client activity

IngestionServiceException - An exception returned from the service

See Also:

ResultSetSourceInfo, IngestionProperties

ingestFromStream

public IngestionResult ingestFromStream(StreamSourceInfo streamSourceInfo,
                                        IngestionProperties ingestionProperties)
                                 throws IngestionClientException,
                                        IngestionServiceException

Description copied from interface: IngestClient

Ingest data from an input stream, into Kusto database.

This method ingests the data from a given input stream, described in streamSourceInfo, into Kusto database, according to the properties mentioned in ingestionProperties

Specified by:

ingestFromStream in interface IngestClient

Parameters:

streamSourceInfo - The specific SourceInfo to be ingested

ingestionProperties - Settings used to customize the ingestion operation

Returns:

IngestionResult object including the ingestion result

Throws:

IngestionClientException - An exception originating from a client activity

IngestionServiceException - An exception returned from the service

See Also:

StreamSourceInfo, IngestionProperties

close

public void close()

Specified by:

close in interface AutoCloseable

Specified by:

close in interface Closeable