KinesisConsumerConfig
Allow for customised low level configuration of the Amazon AWS Client.
Allow for customised low level configuration of the Amazon AWS Client.
Note, specifying a value here may override specified other configurations.
Default is to use the default AWS configuration - this is likely a reasonable value in most cases.
Get the AWS CredentialsProvider for Kinesis, etc., access.
Get the AWS CredentialsProvider for Kinesis, etc., access.
Defaults to com.amazonaws.auth.DefaultAWSCredentialsProviderChain
How long to backoff after a communication, etc., error issuing a Checkpoint on a stream/shard, before automatically retrying.
How long to backoff after a communication, etc., error issuing a Checkpoint on a stream/shard, before automatically retrying.
Defaults to 30 seconds.
How often to retry a checkpoint, on failure, before completely failing.
How often to retry a checkpoint, on failure, before completely failing.
Defaults to 3 attempts.
Create a Kinesis Client Lib Configuration.
Create a Kinesis Client Lib Configuration. This is used internally when configuring the Java Kinesis client connections.
The consumer associated with this confguration will be identified by s"$appName-$streamName" (this includes
the DynamoDB table, etc.)
Allow for overriding the awsCredentialsProvider for connection to Dynamo.
Allow for overriding the awsCredentialsProvider for connection to Dynamo.
This allows the Kinesis stream and lease/checkpoint DynamoDB table to exist in different accounts, etc.
Defaults to None
Idle time between calls to fetch data from Kinesis
Idle time between calls to fetch data from Kinesis
defaults to 10 seconds
One of LATEST or TRIM_HORIZON.
One of LATEST or TRIM_HORIZON. The Amazon Kinesis Client Library will start
fetching records from this position when the application starts up if there are no checkpoints. If there
are checkpoints, we will process records from the checkpoint position.
defaults to TRIM_HORIZON
Optionally explicitly define the endpoint to be used for Kinesis.
Optionally explicitly define the endpoint to be used for Kinesis.
If specified, this setting will be used to configure the Amazon Kinesis client to read from setting, overwriting the configured (as per regionName) region name (but only for Kinesis - the DynamoDB and CloudWatch will still use the configured region name.)
Defaults to None
Lease duration (leases not renewed within this period will be claimed by others)
Lease duration (leases not renewed within this period will be claimed by others)
defaults to 30 seconds
Maximum number of records in a batch.
Maximum number of records in a batch.
defaults to 1000
Metrics are buffered for at most this long before publishing to CloudWatch
Metrics are buffered for at most this long before publishing to CloudWatch
defaults to 20 seconds
Max number of metrics to buffer before publishing to CloudWatch
Max number of metrics to buffer before publishing to CloudWatch
defaults to 1000
Wait for this long between polls to check if parent shards are done
Wait for this long between polls to check if parent shards are done
defaults to 20 seconds
Whether every empty batch should be passed to the application.
Whether every empty batch should be passed to the application.
The underlying client effectively polls the kinesis stream server periodically - when there are no records present to be passed to the application this is effectively an empty batch - by default not all these get passed to the application.
Note, in the case where an application does not checkpoint every batch, empty batches may still
be passed to the application, even if this specifies false - however, in that case, empty batches will only
be passed when there has been records passed since the last checkpoint - if no records have been passed since last
checkpoint then no empty batches will be passed on.
It is not expected, or necessary, to configure this to true - however, it may be useful for certain types of
applications to process
Defaults to false
Specify the Amazon region name to be used.
Specify the Amazon region name to be used.
Defaults to "us-east-1"
Time between tasks to sync leases and Kinesis shards
Time between tasks to sync leases and Kinesis shards
defaults to 1 minute
Backoff period when tasks encounter an exception
Backoff period when tasks encounter an exception
defaults to 20 seconds
Used to identify different worker processes - needs to be different for each instance, not just application.
Used to identify different worker processes - needs to be different for each instance, not just application.
This value is logged (INFO) when a configuration for a stream is created - this is useful to assist inspection of the contents of the associated dynamoDB table.
Defaults to the string value of a random UUID.
Configuration for access around a Kinesis Stream.
There are some (arguably) reasonable defaults here.