software.amazon.kinesis.leases.dynamodb

Class DynamoDBLeaseTaker

java.lang.Object

software.amazon.kinesis.leases.dynamodb.DynamoDBLeaseTaker

All Implemented Interfaces:

LeaseTaker

public class DynamoDBLeaseTaker
extends Object
implements LeaseTaker

An implementation of LeaseTaker that uses DynamoDB via LeaseRefresher.

Constructor Summary

Constructors

Constructor and Description
DynamoDBLeaseTaker(LeaseRefresher leaseRefresher, String workerIdentifier, long leaseDurationMillis, MetricsFactory metricsFactory)

Method Summary

Modifier and Type	Method and Description
List<Lease>	allLeases() Default implementation returns an empty list and concrete implementaion is expected to return all leases for the application that are in the lease table either by reading lease table or from an internal cache.
String	getWorkerIdentifier()
Map<String,Lease>	takeLeases() Compute the set of leases available to be taken and attempt to take them.
DynamoDBLeaseTaker	withMaxLeasesForWorker(int maxLeasesForWorker) Worker will not acquire more than the specified max number of leases even if there are more shards that need to be processed.
DynamoDBLeaseTaker	withMaxLeasesToStealAtOneTime(int maxLeasesToStealAtOneTime) Max leases to steal from a more loaded Worker at one time (for load balancing).
DynamoDBLeaseTaker	withVeryOldLeaseDurationNanosMultipler(long veryOldLeaseDurationNanosMultipler) Overrides the default very old lease duration nanos multiplier to increase the threshold for taking very old leases.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DynamoDBLeaseTaker

public DynamoDBLeaseTaker(LeaseRefresher leaseRefresher,
                          String workerIdentifier,
                          long leaseDurationMillis,
                          MetricsFactory metricsFactory)

Method Detail

withMaxLeasesForWorker

public DynamoDBLeaseTaker withMaxLeasesForWorker(int maxLeasesForWorker)

Worker will not acquire more than the specified max number of leases even if there are more shards that need to be processed. This can be used in scenarios where a worker is resource constrained or to prevent lease thrashing when small number of workers pick up all leases for small amount of time during deployment. Note that setting a low value may cause data loss (e.g. if there aren't enough Workers to make progress on all shards). When setting the value for this property, one must ensure enough workers are present to process shards and should consider future resharding, child shards that may be blocked on parent shards, some workers becoming unhealthy, etc.

Parameters:

maxLeasesForWorker - Max leases this Worker can handle at a time

Returns:

LeaseTaker

withVeryOldLeaseDurationNanosMultipler

public DynamoDBLeaseTaker withVeryOldLeaseDurationNanosMultipler(long veryOldLeaseDurationNanosMultipler)

Overrides the default very old lease duration nanos multiplier to increase the threshold for taking very old leases. Setting this to a higher value than 3 will increase the threshold for very old lease taking.

Parameters:

veryOldLeaseDurationNanosMultipler - Very old lease duration multiplier for adjusting very old lease taking.

Returns:

LeaseTaker

withMaxLeasesToStealAtOneTime

public DynamoDBLeaseTaker withMaxLeasesToStealAtOneTime(int maxLeasesToStealAtOneTime)

Max leases to steal from a more loaded Worker at one time (for load balancing). Setting this to a higher number can allow for faster load convergence (e.g. during deployments, cold starts), but can cause higher churn in the system.

Parameters:

maxLeasesToStealAtOneTime - Steal up to this many leases at one time (for load balancing)

Returns:

LeaseTaker

takeLeases

public Map<String,Lease> takeLeases()
                             throws DependencyException,
                                    InvalidStateException

Compute the set of leases available to be taken and attempt to take them. Lease taking rules are: 1) If a lease's counter hasn't changed in long enough, try to take it. 2) If we see a lease we've never seen before, take it only if owner == null. If it's owned, odds are the owner is holding it. We can't tell until we see it more than once. 3) For load balancing purposes, you may violate rules 1 and 2 for EXACTLY ONE lease per call of takeLeases().

Specified by:

takeLeases in interface LeaseTaker

Returns:

map of shardId to Lease object for leases we just successfully took.

Throws:

DependencyException - on unexpected DynamoDB failures

InvalidStateException - if lease table does not exist

getWorkerIdentifier

public String getWorkerIdentifier()

Specified by:

getWorkerIdentifier in interface LeaseTaker

Returns:

workerIdentifier for this LeaseTaker

allLeases

public List<Lease> allLeases()

Default implementation returns an empty list and concrete implementaion is expected to return all leases for the application that are in the lease table either by reading lease table or from an internal cache.

Specified by:

allLeases in interface LeaseTaker

Returns:

all leases for the application that are in the lease table