Utility functions for mapping ids or indices to a shard. For our purposes, a shard is an instance with a set of server groups. The union of data from all groups comprises a full copy of the overall dataset. To allow for smaller deployment units, an individual group or subset of the groups can be replicated. For redundancy, groups could be replicated all the time. At Netflix, we typically replicate the overall set of server groups in another region or zone instead.
This class specifically focuses on relatively simple sharding schemes where the component making the decision only needs to know the set of instances and a slot for each instance. Edda is one example of a system that provides this information for AWS auto-scaling groups. More complex sharding schemes that require additional infrastructure, e.g, zookeeper, are out of scope here. There are two sharding modes supported by this class:
Mapping an id for a tagged item to a shard. This is typically done while data is flowing into the system and each datapoint can be routed based on the id.
Mapping an positional index to a shard. This is typically done for loading data that has been processed via Hadoop or similar tools and stored in a fixed number of files. There should be a manifest with an order list of the files for a given time and the position can be used to map to a shard. When using this approach it is recommended to use a highly composite number for the number of files. This makes it easier to pick a number of groups and sizes for the groups such that each instance will get the same number of files.
When mapping this to AWS an overall deployment is typically a set of auto-scaling groups (ASG). Each instance should get the same amount of data if possible given the set of files. Deployments are typically done as a red/black push of one ASG at a time. So the amount of additional capacity during a push is the size of one of these groups if deployments across the groups are performed serially. While multiple ASGs for a particular group are active the data will be replicated across them.
case class Group[T](name: String, instances: Array[T])
Group of instances representing a subset of the overall deployment.
Group of instances representing a subset of the overall deployment.
Value parameters
instances
Instances that are part of the group. The order of this array is important to ensure that an instance will always get the same set of data. The position is used to associate data to the instance.
name
Name of the group. In the case of replicas each replica group should have the same name.
Attributes
Supertypes
trait Serializable
trait Product
trait Equals
class Object
trait Matchable
class Any
Show all
class LocalMapper[T](groupSize: Int, instanceIdx: Int, groupIdx: Int, numGroups: Int)
Mapper intended to run on a given instance and check to see if data should be loaded there.
Mapper intended to run on a given instance and check to see if data should be loaded there.
Mapper for finding the instance that should receive data for an id or index of a file. There can be multiple groups with a given name and data will be replicated across all of the groups for that name.
Mapper for finding the instance that should receive data for an id or index of a file. There can be multiple groups with a given name and data will be replicated across all of the groups for that name.
Creates a mapper used to route data to the appropriate instance. This form is typically used as data is flowing into the system when replicas are not a concern. If replication is needed, then see replicaMapper instead.
Creates a mapper used to route data to the appropriate instance. This form is typically used as data is flowing into the system when replicas are not a concern. If replication is needed, then see replicaMapper instead.
Creates a mapper used to route data to the appropriate instance. This form is typically used as data is flowing into the system when replicas are not a concern. If replication is needed, then see replicaMapper instead.
Creates a mapper used to route data to the appropriate instance. This form is typically used as data is flowing into the system when replicas are not a concern. If replication is needed, then see replicaMapper instead.
Value parameters
groups
Set of groups that makes up the complete data set.
Creates a mapper used to route data to the appropriate instance. This form is used as data is flowing into the system and there can be replicas for the groups.
Creates a mapper used to route data to the appropriate instance. This form is used as data is flowing into the system and there can be replicas for the groups.
Value parameters
groups
Set of groups that makes up the complete deployment.