com.amazonaws.services.kinesis.producer

Class KinesisProducer

java.lang.Object

com.amazonaws.services.kinesis.producer.KinesisProducer

All Implemented Interfaces:

IKinesisProducer

public class KinesisProducer
extends Object
implements IKinesisProducer

An interface to the native KPL daemon. This class handles the creation, destruction and use of the child process.

Use a single instance within the application whenever possible:

• One child process is spawned per instance of KinesisProducer. Additional instances introduce overhead and reduce aggregation efficiency.
• All streams within a region that can be accessed with the same credentials can share the same KinesisProducer instance.
• The addUserRecord(java.lang.String, java.lang.String, java.nio.ByteBuffer) methods are thread safe, and be called concurrently.
• Therefore, unless you need to put to multiple regions, or need to use different credentials for different streams, you should avoid creating multiple instances of KinesisProducer.

Author:

chaodeng

Constructor Summary

Constructors

Modifier	Constructor and Description
	KinesisProducer() Start up a KinesisProducer instance.
protected	KinesisProducer(File inPipe, File outPipe) Connect to a running daemon.
	KinesisProducer(KinesisProducerConfiguration config) Start up a KinesisProducer instance.

Method Summary

Modifier and Type	Method and Description
com.google.common.util.concurrent.ListenableFuture<UserRecordResult>	addUserRecord(String stream, String partitionKey, ByteBuffer data) Put a record asynchronously.
com.google.common.util.concurrent.ListenableFuture<UserRecordResult>	addUserRecord(String stream, String partitionKey, String explicitHashKey, ByteBuffer data) Put a record asynchronously.
com.google.common.util.concurrent.ListenableFuture<UserRecordResult>	addUserRecord(String stream, String partitionKey, String explicitHashKey, ByteBuffer data, com.amazonaws.services.schemaregistry.common.Schema schema)
com.google.common.util.concurrent.ListenableFuture<UserRecordResult>	addUserRecord(UserRecord userRecord) Put a record asynchronously.
void	destroy() Immediately kill the child process.
void	flush() Instruct the child process to perform a flush, sending some of the records it has buffered.
void	flush(String stream) Instruct the child process to perform a flush, sending some of the records it has buffered for the specified stream.
void	flushSync() Instructs the child process to flush all records and waits until all records are complete (either succeeding or failing).
List<Metric>	getMetrics() Get metrics from the KPL.
List<Metric>	getMetrics(int windowSeconds) Get metrics from the KPL.
List<Metric>	getMetrics(String metricName) Get metrics from the KPL.
List<Metric>	getMetrics(String metricName, int windowSeconds) Get metrics from the KPL.
long	getOldestRecordTimeInMillis() Get the time in millis for the oldest record currently waiting in kpl to be processed for sending to kinesis endpoint.
int	getOutstandingRecordsCount() Get the number of unfinished records currently being processed.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

KinesisProducer

public KinesisProducer(KinesisProducerConfiguration config)

Start up a KinesisProducer instance.

Since this creates a child process, it is fairly expensive. Avoid creating more than one per application, unless putting to multiple regions at the same time. All streams in the same region can share the same instance.

All methods in KinesisProducer are thread-safe.

Parameters:

config - Configuration for the KinesisProducer. See the docs for that class for details.

See Also:

KinesisProducerConfiguration

KinesisProducer

public KinesisProducer()

Start up a KinesisProducer instance.

Since this creates a child process, it is fairly expensive. Avoid creating more than one per application, unless putting to multiple regions at the same time. All streams in the same region can share the same instance.

The KPL will use a set of default configurations. You can set custom configuration using the constructor that takes a KinesisProducerConfiguration object.

All methods in KinesisProducer are thread-safe.

KinesisProducer

protected KinesisProducer(File inPipe,
                          File outPipe)

Connect to a running daemon. Does not start a child process. Used for testing.

Parameters:

inPipe -

outPipe -

Method Detail

addUserRecord

public com.google.common.util.concurrent.ListenableFuture<UserRecordResult> addUserRecord(String stream,
                                                                                          String partitionKey,
                                                                                          ByteBuffer data)

Put a record asynchronously. A ListenableFuture is returned that can be used to retrieve the result, either by polling or by registering a callback.

The return value can be disregarded if you do not wish to process the result. Under the covers, the KPL will automatically re-attempt puts in case of transient errors (including throttling). A failed result is generally returned only if an irrecoverable error is detected (e.g. trying to put to a stream that doesn't exist), or if the record expires.

Thread safe.

To add a listener to the future:

ListenableFuture<PutRecordResult> f = myKinesisProducer.addUserRecord(...); com.google.common.util.concurrent.Futures.addCallback(f, callback, executor);

where callback is an instance of FutureCallback and executor is an instance of Executor.

Important:

If long-running tasks are performed in the callbacks, it is recommended that a custom executor be provided when registering callbacks to ensure that there are enough threads to achieve the desired level of parallelism. By default, the KPL will use an internal thread pool to execute callbacks, but this pool may not have a sufficient number of threads if a large number is desired.

Another option would be to hand the result off to a different component for processing and keep the callback routine fast.

Specified by:

addUserRecord in interface IKinesisProducer

Parameters:

stream - Stream to put to.

partitionKey - Partition key. Length must be at least one, and at most 256 (inclusive).

data - Binary data of the record. Maximum size 1MiB.

Returns:

A future for the result of the put.

Throws:

IllegalArgumentException - if input does not meet stated constraints

DaemonException - if the child process is dead

See Also:

ListenableFuture, UserRecordResult, KinesisProducerConfiguration.setRecordTtl(long), UserRecordFailedException

addUserRecord

public com.google.common.util.concurrent.ListenableFuture<UserRecordResult> addUserRecord(UserRecord userRecord)

Put a record asynchronously. A ListenableFuture is returned that can be used to retrieve the result, either by polling or by registering a callback.

The return value can be disregarded if you do not wish to process the result. Under the covers, the KPL will automatically re-attempt puts in case of transient errors (including throttling). A failed result is generally returned only if an irrecoverable error is detected (e.g. trying to put to a stream that doesn't exist), or if the record expires.

Thread safe.

To add a listener to the future:

ListenableFuture<PutRecordResult> f = myKinesisProducer.addUserRecord(...); com.google.common.util.concurrent.Futures.addCallback(f, callback, executor);

where callback is an instance of FutureCallback and executor is an instance of Executor.

Important:

If long-running tasks are performed in the callbacks, it is recommended that a custom executor be provided when registering callbacks to ensure that there are enough threads to achieve the desired level of parallelism. By default, the KPL will use an internal thread pool to execute callbacks, but this pool may not have a sufficient number of threads if a large number is desired.

Another option would be to hand the result off to a different component for processing and keep the callback routine fast.

Specified by:

addUserRecord in interface IKinesisProducer

Parameters:

userRecord - All data necessary to write to the stream.

Returns:

A future for the result of the put.

Throws:

IllegalArgumentException - if input does not meet stated constraints

DaemonException - if the child process is dead

See Also:

ListenableFuture, UserRecordResult, KinesisProducerConfiguration.setRecordTtl(long), UserRecordFailedException

addUserRecord

public com.google.common.util.concurrent.ListenableFuture<UserRecordResult> addUserRecord(String stream,
                                                                                          String partitionKey,
                                                                                          String explicitHashKey,
                                                                                          ByteBuffer data)

Put a record asynchronously. A ListenableFuture is returned that can be used to retrieve the result, either by polling or by registering a callback.

The return value can be disregarded if you do not wish to process the result. Under the covers, the KPL will automatically reattempt puts in case of transient errors (including throttling). A failed result is generally returned only if an irrecoverable error is detected (e.g. trying to put to a stream that doesn't exist), or if the record expires.

Thread safe.

To add a listener to the future:

ListenableFuture<PutRecordResult> f = myKinesisProducer.addUserRecord(...); com.google.common.util.concurrent.Futures.addCallback(f, callback, executor);

where callback is an instance of FutureCallback and executor is an instance of Executor.

Important:

If long-running tasks are performed in the callbacks, it is recommended that a custom executor be provided when registering callbacks to ensure that there are enough threads to achieve the desired level of parallelism. By default, the KPL will use an internal thread pool to execute callbacks, but this pool may not have a sufficient number of threads if a large number is desired.

Another option would be to hand the result off to a different component for processing and keep the callback routine fast.

Specified by:

addUserRecord in interface IKinesisProducer

Parameters:

stream - Stream to put to.

partitionKey - Partition key. Length must be at least one, and at most 256 (inclusive).

explicitHashKey - The hash value used to explicitly determine the shard the data record is assigned to by overriding the partition key hash. Must be a valid string representation of a positive integer with value between 0 and 2^128 - 1 (inclusive).

data - Binary data of the record. Maximum size 1MiB.

Returns:

A future for the result of the put.

Throws:

IllegalArgumentException - if input does not meet stated constraints

DaemonException - if the child process is dead

See Also:

ListenableFuture, UserRecordResult, KinesisProducerConfiguration.setRecordTtl(long), UserRecordFailedException

addUserRecord

public com.google.common.util.concurrent.ListenableFuture<UserRecordResult> addUserRecord(String stream,
                                                                                          String partitionKey,
                                                                                          String explicitHashKey,
                                                                                          ByteBuffer data,
                                                                                          com.amazonaws.services.schemaregistry.common.Schema schema)

Specified by:

addUserRecord in interface IKinesisProducer

getOutstandingRecordsCount

public int getOutstandingRecordsCount()

Get the number of unfinished records currently being processed. The records could either be waiting to be sent to the child process, or have reached the child process and are being worked on.

This is equal to the number of futures returned from addUserRecord(java.lang.String, java.lang.String, java.nio.ByteBuffer) that have not finished.

Specified by:

getOutstandingRecordsCount in interface IKinesisProducer

Returns:

The number of unfinished records currently being processed.

getOldestRecordTimeInMillis

public long getOldestRecordTimeInMillis()

Get the time in millis for the oldest record currently waiting in kpl to be processed for sending to kinesis endpoint. The records could either be waiting to be sent to the child process, or have reached the child process and are being worked on.

This is time in millis from the oldest future submitted from addUserRecord(java.lang.String, java.lang.String, java.nio.ByteBuffer) that have not finished. This returns 0 if there are no pending records in processing state.

Specified by:

getOldestRecordTimeInMillis in interface IKinesisProducer

Returns:

The time in millis since the oldest record is pending to be sent to kinesis endpoint. Returns 0 if there are no records in processing state.

getMetrics

public List<Metric> getMetrics(String metricName,
                               int windowSeconds)
                        throws InterruptedException,
                               ExecutionException

Get metrics from the KPL.

The KPL computes and buffers useful metrics. Use this method to retrieve them. The same metrics are also uploaded to CloudWatch (unless disabled).

Multiple metrics exist for the same name, each with a different list of dimensions (e.g. stream name). This method will fetch all metrics with the provided name.

See the stand-alone metrics documentation for details about individual metrics.

This method is synchronous and will block while the data is being retrieved.

Specified by:

getMetrics in interface IKinesisProducer

Parameters:

metricName - Name of the metrics to fetch.

windowSeconds - Fetch data from the last N seconds. The KPL maintains data at per second granularity for the past minute. To get total cumulative data since the start of the program, use the overloads that do not take this argument.

Returns:

A list of metrics with the provided name.

Throws:

ExecutionException - If an error occurred while fetching metrics from the child process.

InterruptedException - If the thread is interrupted while waiting for the response from the child process.

See Also:

Metric

getMetrics

public List<Metric> getMetrics(String metricName)
                        throws InterruptedException,
                               ExecutionException

Get metrics from the KPL.

The KPL computes and buffers useful metrics. Use this method to retrieve them. The same metrics are also uploaded to CloudWatch (unless disabled).

Multiple metrics exist for the same name, each with a different list of dimensions (e.g. stream name). This method will fetch all metrics with the provided name.

The retrieved data represents cumulative statistics since the start of the program. To get data from a smaller window, use getMetrics(String, int).

See the stand-alone metrics documentation for details about individual metrics.

This method is synchronous and will block while the data is being retrieved.

Specified by:

getMetrics in interface IKinesisProducer

Parameters:

metricName - Name of the metrics to fetch.

Returns:

A list of metrics with the provided name.

Throws:

ExecutionException - If an error occurred while fetching metrics from the child process.

InterruptedException - If the thread is interrupted while waiting for the response from the child process.

See Also:

Metric

getMetrics

public List<Metric> getMetrics()
                        throws InterruptedException,
                               ExecutionException

Get metrics from the KPL.

The KPL computes and buffers useful metrics. Use this method to retrieve them. The same metrics are also uploaded to CloudWatch (unless disabled).

This method fetches all metrics available. To fetch only metrics with a given name, use getMetrics(String).

The retrieved data represents cumulative statistics since the start of the program. To get data from a smaller window, use getMetrics(int).

See the stand-alone metrics documentation for details about individual metrics.

This method is synchronous and will block while the data is being retrieved.

Specified by:

getMetrics in interface IKinesisProducer

Returns:

A list of all of the metrics maintained by the KPL.

Throws:

ExecutionException - If an error occurred while fetching metrics from the child process.

InterruptedException - If the thread is interrupted while waiting for the response from the child process.

See Also:

Metric

getMetrics

public List<Metric> getMetrics(int windowSeconds)
                        throws InterruptedException,
                               ExecutionException

Get metrics from the KPL.

The KPL computes and buffers useful metrics. Use this method to retrieve them. The same metrics are also uploaded to CloudWatch (unless disabled).

This method fetches all metrics available. To fetch only metrics with a given name, use getMetrics(String, int).

See the stand-alone metrics documentation for details about individual metrics.

This method is synchronous and will block while the data is being retrieved.

Specified by:

getMetrics in interface IKinesisProducer

Parameters:

windowSeconds - Fetch data from the last N seconds. The KPL maintains data at per second granularity for the past minute. To get total cumulative data since the start of the program, use the overloads that do not take this argument.

Returns:

A list of metrics with the provided name.

Throws:

ExecutionException - If an error occurred while fetching metrics from the child process.

InterruptedException - If the thread is interrupted while waiting for the response from the child process.

See Also:

Metric

destroy

public void destroy()

Immediately kill the child process. This will cause all outstanding futures to fail immediately.

To perform a graceful shutdown instead, there are several options:

• Call flush() and wait (perhaps with a time limit) for all futures. If you were sending a very high volume of data you may need to call flush multiple times to clear all buffers.
• Poll getOutstandingRecordsCount() until it returns 0.
• Call flushSync(), which blocks until completion.

Once all records are confirmed with one of the above, call destroy to terminate the child process. If you are terminating the JVM then calling destroy is unnecessary since it will be done automatically.

Specified by:

destroy in interface IKinesisProducer

flush

public void flush(String stream)

Instruct the child process to perform a flush, sending some of the records it has buffered for the specified stream.

This does not guarantee that all buffered records will be sent, only that most of them will; to flush all records and wait for completion, use flushSync().

This method returns immediately without blocking.

Specified by:

flush in interface IKinesisProducer

Parameters:

stream - Stream to flush

Throws:

DaemonException - if the child process is dead

flush

public void flush()

Instruct the child process to perform a flush, sending some of the records it has buffered. Applies to all streams.

This does not guarantee that all buffered records will be sent, only that most of them will; to flush all records and wait for completion, use flushSync().

This method returns immediately without blocking.

Specified by:

flush in interface IKinesisProducer

Throws:

DaemonException - if the child process is dead

flushSync

public void flushSync()

Instructs the child process to flush all records and waits until all records are complete (either succeeding or failing).

The wait includes any retries that need to be performed. Depending on your configuration of record TTL and request timeout, this can potentially take a long time if the library is having trouble delivering records to the backend, for example due to network problems.

This is useful if you need to shutdown your application and want to make sure all records are delivered before doing so.

Specified by:

flushSync in interface IKinesisProducer

Throws:

DaemonException - if the child process is dead

See Also:

KinesisProducerConfiguration.setRecordTtl(long), KinesisProducerConfiguration.setRequestTimeout(long)