Shards
Related Doc:
package util
Group of instances representing a subset of the overall deployment.
Group of instances representing a subset of the overall deployment.
Name of the group. In the case of replicas each replica group should have the same name.
Instances that are part of the group. The order of this array is important to ensure that an instance will always get the same set of data. The position is used to associate data to the instance.
Mapper intended to run on a given instance and check to see if data should be loaded there.
Mapper for finding the instance that should receive data for an id or index of a file.
Mapper for finding the instance that should receive data for an id or index of a file.
Mapper for finding the instance that should receive data for an id or index of a file. There can be multiple groups with a given name and data will be replicated across all of the groups for that name.
Creates a mapper that can be used on an instance of a group.
Creates a mapper that can be used on an instance of a group. This is typically used if the local instance needs to figure out what data to load.
Size of the group that contains the instance.
Index for this instance within the group.
Index of this group within the overall set of groups.
Number of groups that make up the complete deployment.
Mapper
Creates a mapper used to route data to the appropriate instance.
Creates a mapper used to route data to the appropriate instance. This form is typically
used as data is flowing into the system when replicas are not a concern. If replication
is needed, then see replicaMapper instead.
Set of groups that makes up the complete data set.
Mapper for routing data to instances.
Creates a mapper used to route data to the appropriate instance.
Creates a mapper used to route data to the appropriate instance. This form is typically
used as data is flowing into the system when replicas are not a concern. If replication
is needed, then see replicaMapper instead.
Single group that makes up the complete data set.
Mapper for routing data to instances.
Creates a mapper used to route data to the appropriate instance.
Creates a mapper used to route data to the appropriate instance. This form is used as data is flowing into the system and there can be replicas for the groups.
Set of groups that makes up the complete deployment.
Mapper for routing data to instances.
(Since version ) see corresponding Javadoc for more information.
Utility functions for mapping ids or indices to a shard. For our purposes, a shard is an instance with a set of server groups. The union of data from all groups comprises a full copy of the overall dataset. To allow for smaller deployment units, an individual group or subset of the groups can be replicated. For redundancy, groups could be replicated all the time. At Netflix, we typically replicate the overall set of server groups in another region or zone instead.
This class specifically focuses on relatively simple sharding schemes where the component making the decision only needs to know the set of instances and a slot for each instance. Edda is one example of a system that provides this information for AWS auto-scaling groups. More complex sharding schemes that require additional infrastructure, e.g, zookeeper, are out of scope here. There are two sharding modes supported by this class:
1. Mapping an id for a tagged item to a shard. This is typically done while data is flowing into the system and each datapoint can be routed based on the id.
2. Mapping an positional index to a shard. This is typically done for loading data that has been processed via Hadoop or similar tools and stored in a fixed number of files. There should be a manifest with an order list of the files for a given time and the position can be used to map to a shard. When using this approach it is recommended to use a [highly composite number][hcn] for the number of files. This makes it easier to pick a number of groups and sizes for the groups such that each instance will get the same number of files.
When mapping this to AWS an overall deployment is typically a set of auto-scaling groups (ASG). Each instance should get the same amount of data if possible given the set of files. Deployments are typically done as a red/black push of one ASG at a time. So the amount of additional capacity during a push is the size of one of these groups if deployments across the groups are performed serially. While multiple ASGs for a particular group are active the data will be replicated across them.
[hcn]: https://en.wikipedia.org/wiki/Highly_composite_number