software.amazon.kinesis.checkpoint

Class ShardRecordProcessorCheckpointer

java.lang.Object

software.amazon.kinesis.checkpoint.ShardRecordProcessorCheckpointer

All Implemented Interfaces:

RecordProcessorCheckpointer

public class ShardRecordProcessorCheckpointer
extends Object
implements RecordProcessorCheckpointer

This class is used to enable RecordProcessors to checkpoint their progress. The Amazon Kinesis Client Library will instantiate an object and provide a reference to the application ShardRecordProcessor instance. Amazon Kinesis Client Library will create one instance per shard assignment.

Constructor Summary

Constructors

Constructor and Description
ShardRecordProcessorCheckpointer()

Method Summary

Modifier and Type	Method and Description
void	checkpoint() This method will checkpoint the progress at the last data record that was delivered to the record processor.
void	checkpoint(software.amazon.awssdk.services.kinesis.model.Record record) This method will checkpoint the progress at the provided record.
void	checkpoint(String sequenceNumber) This method will checkpoint the progress at the provided sequenceNumber.
void	checkpoint(String sequenceNumber, long subSequenceNumber) This method will checkpoint the progress at the provided sequenceNumber and subSequenceNumber, the latter for aggregated records produced with the Producer Library.
void	largestPermittedCheckpointValue(ExtendedSequenceNumber largestPermittedCheckpointValue)
PreparedCheckpointer	prepareCheckpoint() This method will record a pending checkpoint at the last data record that was delivered to the record processor.
PreparedCheckpointer	prepareCheckpoint(byte[] applicationState) This method will record a pending checkpoint at the last data record that was delivered to the record processor.
PreparedCheckpointer	prepareCheckpoint(software.amazon.awssdk.services.kinesis.model.Record record) This method will record a pending checkpoint at the at the provided record.
PreparedCheckpointer	prepareCheckpoint(software.amazon.awssdk.services.kinesis.model.Record record, byte[] applicationState) This method will record a pending checkpoint at the at the provided record.
PreparedCheckpointer	prepareCheckpoint(String sequenceNumber) This method will record a pending checkpoint at the provided sequenceNumber.
PreparedCheckpointer	prepareCheckpoint(String sequenceNumber, byte[] applicationState) This method will record a pending checkpoint at the provided sequenceNumber.
PreparedCheckpointer	prepareCheckpoint(String sequenceNumber, long subSequenceNumber) This method will record a pending checkpoint at the provided sequenceNumber and subSequenceNumber, the latter for aggregated records produced with the Producer Library.
PreparedCheckpointer	prepareCheckpoint(String sequenceNumber, long subSequenceNumber, byte[] applicationState) This method will record a pending checkpoint at the provided sequenceNumber and subSequenceNumber, the latter for aggregated records produced with the Producer Library.
void	sequenceNumberAtShardEnd(ExtendedSequenceNumber extendedSequenceNumber) Used to remember the last extended sequence number before SHARD_END to allow us to prevent the checkpointer from checkpointing at the end of the shard twice (i.e.
void	setInitialCheckpointValue(ExtendedSequenceNumber initialCheckpoint)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface software.amazon.kinesis.processor.RecordProcessorCheckpointer

checkpointer

Constructor Detail

ShardRecordProcessorCheckpointer

public ShardRecordProcessorCheckpointer()

Method Detail

checkpoint

public void checkpoint()
                throws KinesisClientLibDependencyException,
                       InvalidStateException,
                       ThrottlingException,
                       ShutdownException

This method will checkpoint the progress at the last data record that was delivered to the record processor. Upon fail over (after a successful checkpoint() call), the new/replacement ShardRecordProcessor instance will receive data records whose sequenceNumber > checkpoint position (for each partition key). In steady state, applications should checkpoint periodically (e.g. once every 5 minutes). Calling this API too frequently can slow down the application (because it puts pressure on the underlying checkpoint storage layer).

Specified by:

checkpoint in interface RecordProcessorCheckpointer

Throws:

KinesisClientLibDependencyException - Encountered an issue when storing the checkpoint. The application can backoff and retry.

InvalidStateException - Can't store checkpoint. Unable to store the checkpoint in the DynamoDB table (e.g. table doesn't exist).

ThrottlingException - Can't store checkpoint. Can be caused by checkpointing too frequently. Consider increasing the throughput/capacity of the checkpoint store or reducing checkpoint frequency.

ShutdownException - The record processor instance has been shutdown. Another instance may have started processing some of these records already. The application should abort processing via this ShardRecordProcessor instance.

checkpoint

public void checkpoint(software.amazon.awssdk.services.kinesis.model.Record record)
                throws KinesisClientLibDependencyException,
                       InvalidStateException,
                       ThrottlingException,
                       ShutdownException,
                       IllegalArgumentException

This method will checkpoint the progress at the provided record. This method is analogous to RecordProcessorCheckpointer.checkpoint() but provides the ability to specify the record at which to checkpoint.

Specified by:

checkpoint in interface RecordProcessorCheckpointer

Parameters:

record - A record at which to checkpoint in this shard. Upon failover, the Kinesis Client Library will start fetching records after this record's sequence number.

Throws:

KinesisClientLibDependencyException - Encountered an issue when storing the checkpoint. The application can backoff and retry.

InvalidStateException - Can't store checkpoint. Unable to store the checkpoint in the DynamoDB table (e.g. table doesn't exist).

ThrottlingException - Can't store checkpoint. Can be caused by checkpointing too frequently. Consider increasing the throughput/capacity of the checkpoint store or reducing checkpoint frequency.

ShutdownException - The record processor instance has been shutdown. Another instance may have started processing some of these records already. The application should abort processing via this ShardRecordProcessor instance.

IllegalArgumentException

checkpoint

public void checkpoint(String sequenceNumber)
                throws KinesisClientLibDependencyException,
                       InvalidStateException,
                       ThrottlingException,
                       ShutdownException,
                       IllegalArgumentException

This method will checkpoint the progress at the provided sequenceNumber. This method is analogous to RecordProcessorCheckpointer.checkpoint() but provides the ability to specify the sequence number at which to checkpoint.

Specified by:

checkpoint in interface RecordProcessorCheckpointer

Parameters:

sequenceNumber - A sequence number at which to checkpoint in this shard. Upon failover, the Kinesis Client Library will start fetching records after this sequence number.

Throws:

KinesisClientLibDependencyException - Encountered an issue when storing the checkpoint. The application can backoff and retry.

InvalidStateException - Can't store checkpoint. Unable to store the checkpoint in the DynamoDB table (e.g. table doesn't exist).

ThrottlingException - Can't store checkpoint. Can be caused by checkpointing too frequently. Consider increasing the throughput/capacity of the checkpoint store or reducing checkpoint frequency.

ShutdownException - The record processor instance has been shutdown. Another instance may have started processing some of these records already. The application should abort processing via this ShardRecordProcessor instance.

IllegalArgumentException - The sequence number is invalid for one of the following reasons: 1.) It appears to be out of range, i.e. it is smaller than the last check point value, or larger than the greatest sequence number seen by the associated record processor. 2.) It is not a valid sequence number for a record in this shard.

checkpoint

public void checkpoint(String sequenceNumber,
                       long subSequenceNumber)
                throws KinesisClientLibDependencyException,
                       InvalidStateException,
                       ThrottlingException,
                       ShutdownException,
                       IllegalArgumentException

This method will checkpoint the progress at the provided sequenceNumber and subSequenceNumber, the latter for aggregated records produced with the Producer Library. This method is analogous to RecordProcessorCheckpointer.checkpoint() but provides the ability to specify the sequence and subsequence numbers at which to checkpoint.

Specified by:

checkpoint in interface RecordProcessorCheckpointer

Parameters:

sequenceNumber - A sequence number at which to checkpoint in this shard. Upon failover, the Kinesis Client Library will start fetching records after the given sequence and subsequence numbers.

subSequenceNumber - A subsequence number at which to checkpoint within this shard. Upon failover, the Kinesis Client Library will start fetching records after the given sequence and subsequence numbers.

Throws:

KinesisClientLibDependencyException - Encountered an issue when storing the checkpoint. The application can backoff and retry.

InvalidStateException - Can't store checkpoint. Unable to store the checkpoint in the DynamoDB table (e.g. table doesn't exist).

ThrottlingException - Can't store checkpoint. Can be caused by checkpointing too frequently. Consider increasing the throughput/capacity of the checkpoint store or reducing checkpoint frequency.

ShutdownException - The record processor instance has been shutdown. Another instance may have started processing some of these records already. The application should abort processing via this ShardRecordProcessor instance.

IllegalArgumentException - The sequence number is invalid for one of the following reasons: 1.) It appears to be out of range, i.e. it is smaller than the last check point value, or larger than the greatest sequence number seen by the associated record processor. 2.) It is not a valid sequence number for a record in this shard.

prepareCheckpoint

public PreparedCheckpointer prepareCheckpoint()
                                       throws KinesisClientLibDependencyException,
                                              InvalidStateException,
                                              ThrottlingException,
                                              ShutdownException

This method will record a pending checkpoint at the last data record that was delivered to the record processor. If the application fails over between calling prepareCheckpoint() and checkpoint(), the init() method of the next IRecordProcessor for this shard will be informed of the prepared sequence number Application should use this to assist with idempotency across failover by calling prepareCheckpoint before having side effects, then by calling checkpoint on the returned PreparedCheckpointer after side effects are complete. Use the sequence number passed in to init() to behave idempotently.

Specified by:

prepareCheckpoint in interface RecordProcessorCheckpointer

Returns:

an PreparedCheckpointer object that can be called later to persist the checkpoint.

Throws:

KinesisClientLibDependencyException - Encountered an issue when storing the pending checkpoint. The application can backoff and retry.

InvalidStateException - Can't store pending checkpoint. Unable to store the checkpoint in the DynamoDB table (e.g. table doesn't exist).

ThrottlingException - Can't store pending checkpoint. Can be caused by checkpointing too frequently. Consider increasing the throughput/capacity of the checkpoint store or reducing checkpoint frequency.

ShutdownException - The record processor instance has been shutdown. Another instance may have started processing some of these records already. The application should abort processing via this ShardRecordProcessor instance.

prepareCheckpoint

public PreparedCheckpointer prepareCheckpoint(byte[] applicationState)
                                       throws KinesisClientLibDependencyException,
                                              InvalidStateException,
                                              ThrottlingException,
                                              ShutdownException

This method will record a pending checkpoint at the last data record that was delivered to the record processor. If the application fails over between calling prepareCheckpoint() and checkpoint(), the init() method of the next IRecordProcessor for this shard will be informed of the prepared sequence number and application state. Application should use this to assist with idempotency across failover by calling prepareCheckpoint before having side effects, then by calling checkpoint on the returned PreparedCheckpointer after side effects are complete. Use the sequence number and application state passed in to init() to behave idempotently.

Specified by:

prepareCheckpoint in interface RecordProcessorCheckpointer

Parameters:

applicationState - arbitrary application state that will be passed to the next record processor that processes the shard.

Returns:

an PreparedCheckpointer object that can be called later to persist the checkpoint.

Throws:

KinesisClientLibDependencyException - Encountered an issue when storing the pending checkpoint. The application can backoff and retry.

InvalidStateException - Can't store pending checkpoint. Unable to store the checkpoint in the DynamoDB table (e.g. table doesn't exist).

ThrottlingException - Can't store pending checkpoint. Can be caused by checkpointing too frequently. Consider increasing the throughput/capacity of the checkpoint store or reducing checkpoint frequency.

ShutdownException - The record processor instance has been shutdown. Another instance may have started processing some of these records already. The application should abort processing via this ShardRecordProcessor instance.

prepareCheckpoint

public PreparedCheckpointer prepareCheckpoint(software.amazon.awssdk.services.kinesis.model.Record record,
                                              byte[] applicationState)
                                       throws KinesisClientLibDependencyException,
                                              InvalidStateException,
                                              ThrottlingException,
                                              ShutdownException

This method will record a pending checkpoint at the at the provided record. This method is analogous to RecordProcessorCheckpointer.prepareCheckpoint() but provides the ability to specify the record and application state at which to prepare the checkpoint.

Specified by:

prepareCheckpoint in interface RecordProcessorCheckpointer

Parameters:

record - A record at which to prepare checkpoint in this shard.

applicationState - arbitrary application state that will be passed to the next record processor that processes the shard. Application should use this to assist with idempotency across failover by calling prepareCheckpoint before having side effects, then by calling checkpoint on the returned PreparedCheckpointer after side effects are complete. Use the sequence number and application state passed in to init() to behave idempotently.

Returns:

an PreparedCheckpointer object that can be called later to persist the checkpoint.

Throws:

KinesisClientLibDependencyException - Encountered an issue when storing the pending checkpoint. The application can backoff and retry.

InvalidStateException - Can't store pending checkpoint. Unable to store the checkpoint in the DynamoDB table (e.g. table doesn't exist).

ThrottlingException - Can't store pending checkpoint. Can be caused by checkpointing too frequently. Consider increasing the throughput/capacity of the checkpoint store or reducing checkpoint frequency.

ShutdownException - The record processor instance has been shutdown. Another instance may have started processing some of these records already. The application should abort processing via this ShardRecordProcessor instance.

prepareCheckpoint

public PreparedCheckpointer prepareCheckpoint(software.amazon.awssdk.services.kinesis.model.Record record)
                                       throws KinesisClientLibDependencyException,
                                              InvalidStateException,
                                              ThrottlingException,
                                              ShutdownException

This method will record a pending checkpoint at the at the provided record. This method is analogous to RecordProcessorCheckpointer.prepareCheckpoint() but provides the ability to specify the record at which to prepare the checkpoint.

Specified by:

prepareCheckpoint in interface RecordProcessorCheckpointer

Parameters:

record - A record at which to prepare checkpoint in this shard. Application should use this to assist with idempotency across failover by calling prepareCheckpoint before having side effects, then by calling checkpoint on the returned PreparedCheckpointer after side effects are complete. Use the sequence number and application state passed in to init() to behave idempotently.

Returns:

an PreparedCheckpointer object that can be called later to persist the checkpoint.

Throws:

KinesisClientLibDependencyException - Encountered an issue when storing the pending checkpoint. The application can backoff and retry.

InvalidStateException - Can't store pending checkpoint. Unable to store the checkpoint in the DynamoDB table (e.g. table doesn't exist).

ThrottlingException - Can't store pending checkpoint. Can be caused by checkpointing too frequently. Consider increasing the throughput/capacity of the checkpoint store or reducing checkpoint frequency.

ShutdownException - The record processor instance has been shutdown. Another instance may have started processing some of these records already. The application should abort processing via this ShardRecordProcessor instance.

prepareCheckpoint

public PreparedCheckpointer prepareCheckpoint(String sequenceNumber)
                                       throws KinesisClientLibDependencyException,
                                              InvalidStateException,
                                              ThrottlingException,
                                              ShutdownException

This method will record a pending checkpoint at the provided sequenceNumber. This method is analogous to RecordProcessorCheckpointer.prepareCheckpoint() but provides the ability to specify the sequence number at which to checkpoint.

Specified by:

prepareCheckpoint in interface RecordProcessorCheckpointer

Parameters:

sequenceNumber - A sequence number at which to prepare checkpoint in this shard.

Returns:

an PreparedCheckpointer object that can be called later to persist the checkpoint.

Throws:

KinesisClientLibDependencyException - Encountered an issue when storing the pending checkpoint. The application can backoff and retry.

InvalidStateException - Can't store pending checkpoint. Unable to store the checkpoint in the DynamoDB table (e.g. table doesn't exist).

ThrottlingException - Can't store pending checkpoint. Can be caused by checkpointing too frequently. Consider increasing the throughput/capacity of the checkpoint store or reducing checkpoint frequency.

ShutdownException - The record processor instance has been shutdown. Another instance may have started processing some of these records already. The application should abort processing via this ShardRecordProcessor instance.

prepareCheckpoint

public PreparedCheckpointer prepareCheckpoint(String sequenceNumber,
                                              byte[] applicationState)
                                       throws KinesisClientLibDependencyException,
                                              InvalidStateException,
                                              ThrottlingException,
                                              ShutdownException,
                                              IllegalArgumentException

This method will record a pending checkpoint at the provided sequenceNumber. This method is analogous to RecordProcessorCheckpointer.prepareCheckpoint() but provides the ability to specify the sequence number and application state at which to checkpoint.

Specified by:

prepareCheckpoint in interface RecordProcessorCheckpointer

Parameters:

sequenceNumber - A sequence number at which to prepare checkpoint in this shard.

applicationState - arbitrary application state that will be passed to the next record processor that processes the shard.

Returns:

an PreparedCheckpointer object that can be called later to persist the checkpoint.

Throws:

KinesisClientLibDependencyException - Encountered an issue when storing the pending checkpoint. The application can backoff and retry.

InvalidStateException - Can't store pending checkpoint. Unable to store the checkpoint in the DynamoDB table (e.g. table doesn't exist).

ThrottlingException - Can't store pending checkpoint. Can be caused by checkpointing too frequently. Consider increasing the throughput/capacity of the checkpoint store or reducing checkpoint frequency.

ShutdownException - The record processor instance has been shutdown. Another instance may have started processing some of these records already. The application should abort processing via this ShardRecordProcessor instance.

IllegalArgumentException - The sequence number is invalid for one of the following reasons: 1.) It appears to be out of range, i.e. it is smaller than the last check point value, or larger than the greatest sequence number seen by the associated record processor. 2.) It is not a valid sequence number for a record in this shard.

prepareCheckpoint

public PreparedCheckpointer prepareCheckpoint(String sequenceNumber,
                                              long subSequenceNumber)
                                       throws KinesisClientLibDependencyException,
                                              InvalidStateException,
                                              ThrottlingException,
                                              ShutdownException

This method will record a pending checkpoint at the provided sequenceNumber and subSequenceNumber, the latter for aggregated records produced with the Producer Library. This method is analogous to RecordProcessorCheckpointer.prepareCheckpoint() but provides the ability to specify the sequence number at which to checkpoint

Specified by:

prepareCheckpoint in interface RecordProcessorCheckpointer

Parameters:

sequenceNumber - A sequence number at which to prepare checkpoint in this shard.

subSequenceNumber - A subsequence number at which to prepare checkpoint within this shard.

Returns:

an PreparedCheckpointer object that can be called later to persist the checkpoint.

Throws:

KinesisClientLibDependencyException - Encountered an issue when storing the pending checkpoint. The application can backoff and retry.

InvalidStateException - Can't store pending checkpoint. Unable to store the checkpoint in the DynamoDB table (e.g. table doesn't exist).

ThrottlingException - Can't store pending checkpoint. Can be caused by checkpointing too frequently. Consider increasing the throughput/capacity of the checkpoint store or reducing checkpoint frequency.

ShutdownException - The record processor instance has been shutdown. Another instance may have started processing some of these records already. The application should abort processing via this ShardRecordProcessor instance.

prepareCheckpoint

public PreparedCheckpointer prepareCheckpoint(String sequenceNumber,
                                              long subSequenceNumber,
                                              byte[] applicationState)
                                       throws KinesisClientLibDependencyException,
                                              InvalidStateException,
                                              ThrottlingException,
                                              ShutdownException,
                                              IllegalArgumentException

This method will record a pending checkpoint at the provided sequenceNumber and subSequenceNumber, the latter for aggregated records produced with the Producer Library. This method is analogous to RecordProcessorCheckpointer.prepareCheckpoint() but provides the ability to specify the sequence number, subsequence number, and application state at which to checkpoint.

Specified by:

prepareCheckpoint in interface RecordProcessorCheckpointer

Parameters:

sequenceNumber - A sequence number at which to prepare checkpoint in this shard.

subSequenceNumber - A subsequence number at which to prepare checkpoint within this shard.

applicationState - arbitrary application state that will be passed to the next record processor that processes the shard.

Returns:

an PreparedCheckpointer object that can be called later to persist the checkpoint.

Throws:

KinesisClientLibDependencyException - Encountered an issue when storing the pending checkpoint. The application can backoff and retry.

InvalidStateException - Can't store pending checkpoint. Unable to store the checkpoint in the DynamoDB table (e.g. table doesn't exist).

ThrottlingException - Can't store pending checkpoint. Can be caused by checkpointing too frequently. Consider increasing the throughput/capacity of the checkpoint store or reducing checkpoint frequency.

ShutdownException - The record processor instance has been shutdown. Another instance may have started processing some of these records already. The application should abort processing via this ShardRecordProcessor instance.

IllegalArgumentException - The sequence number is invalid for one of the following reasons: 1.) It appears to be out of range, i.e. it is smaller than the last check point value, or larger than the greatest sequence number seen by the associated record processor. 2.) It is not a valid sequence number for a record in this shard.

setInitialCheckpointValue

public void setInitialCheckpointValue(ExtendedSequenceNumber initialCheckpoint)

largestPermittedCheckpointValue

public void largestPermittedCheckpointValue(ExtendedSequenceNumber largestPermittedCheckpointValue)

Parameters:

largestPermittedCheckpointValue - the largest permitted checkpoint

sequenceNumberAtShardEnd

public void sequenceNumberAtShardEnd(ExtendedSequenceNumber extendedSequenceNumber)

Used to remember the last extended sequence number before SHARD_END to allow us to prevent the checkpointer from checkpointing at the end of the shard twice (i.e. at the last extended sequence number and then again at SHARD_END).

Parameters:

extendedSequenceNumber -