Package org.opensearch.index

Class TieredMergePolicyProvider

java.lang.Object
org.opensearch.index.TieredMergePolicyProvider
All Implemented Interfaces:
MergePolicyProvider
public final class TieredMergePolicyProvider extends Object implements MergePolicyProvider
A shard in opensearch is a Lucene index, and a Lucene index is broken down into segments. Segments are internal storage elements in the index where the index data is stored, and are immutable up to delete markers. Segments are, periodically, merged into larger segments to keep the index size at bay and expunge deletes. This class customizes and exposes 2 merge policies from lucene - LogByteSizeMergePolicy and TieredMergePolicy.

Tiered merge policy select segments of approximately equal size, subject to an allowed number of segments per tier. The merge policy is able to merge non-adjacent segments, and separates how many segments are merged at once from how many segments are allowed per tier. It also does not over-merge (i.e., cascade merges).

All merge policy settings are dynamic and can be updated on a live index. The merge policy has the following settings:

  • index.merge.policy.expunge_deletes_allowed: When expungeDeletes is called, we only merge away a segment if its delete percentage is over this threshold. Default is 10.
  • index.merge.policy.floor_segment: Segments smaller than this are "rounded up" to this size, i.e. treated as equal (floor) size for merge selection. This is to prevent frequent flushing of tiny segments, thus preventing a long tail in the index. Default is 2mb.
  • index.merge.policy.max_merge_at_once: Maximum number of segments to be merged at a time during "normal" merging. Default is 10.
  • index.merge.policy.max_merged_segment: Maximum sized segment to produce during normal merging (not explicit force merge). This setting is approximate: the estimate of the merged segment size is made by summing sizes of to-be-merged segments (compensating for percent deleted docs). Default is 5gb.
  • index.merge.policy.segments_per_tier: Sets the allowed number of segments per tier. Smaller values mean more merging but fewer segments. Default is 10. Note, this value needs to be >= than the max_merge_at_once otherwise you'll force too many merges to occur.
  • index.merge.policy.deletes_pct_allowed: Controls the maximum percentage of deleted documents that is tolerated in the index. Lower values make the index more space efficient at the expense of increased CPU and I/O activity. Values must be between 5 and 50. Default value is 20.

For normal merging, the policy first computes a "budget" of how many segments are allowed to be in the index. If the index is over-budget, then the policy sorts segments by decreasing size (proportionally considering percent deletes), and then finds the least-cost merge. Merge cost is measured by a combination of the "skew" of the merge (size of largest seg divided by smallest seg), total merge size and pct deletes reclaimed, so that merges with lower skew, smaller size and those reclaiming more deletes, are favored.

If a merge will produce a segment that's larger than max_merged_segment then the policy will merge fewer segments (down to 1 at once, if that one has deletions) to keep the segment size under budget.

Note, this can mean that for large shards that holds many gigabytes of data, the default of max_merged_segment (5gb) can cause for many segments to be in an index, and causing searches to be slower. Use the indices segments API to see the segments that an index has, and possibly either increase the max_merged_segment or issue an optimize call for the index (try and aim to issue it on a low traffic time).

Opensearch.internal:

  • Field Details

    • DEFAULT_EXPUNGE_DELETES_ALLOWED

      public static final double DEFAULT_EXPUNGE_DELETES_ALLOWED
      See Also:

    • DEFAULT_FLOOR_SEGMENT

      public static final org.opensearch.core.common.unit.ByteSizeValue DEFAULT_FLOOR_SEGMENT

    • DEFAULT_MAX_MERGE_AT_ONCE

      public static final int DEFAULT_MAX_MERGE_AT_ONCE
      See Also:

    • DEFAULT_MAX_MERGED_SEGMENT

      public static final org.opensearch.core.common.unit.ByteSizeValue DEFAULT_MAX_MERGED_SEGMENT

    • DEFAULT_SEGMENTS_PER_TIER

      public static final double DEFAULT_SEGMENTS_PER_TIER
      See Also:

    • DEFAULT_RECLAIM_DELETES_WEIGHT

      public static final double DEFAULT_RECLAIM_DELETES_WEIGHT
      See Also:

    • DEFAULT_DELETES_PCT_ALLOWED

      public static final double DEFAULT_DELETES_PCT_ALLOWED
      See Also:

    • INDEX_COMPOUND_FORMAT_SETTING

      public static final Setting<Double> INDEX_COMPOUND_FORMAT_SETTING

    • INDEX_MERGE_POLICY_EXPUNGE_DELETES_ALLOWED_SETTING

      public static final Setting<Double> INDEX_MERGE_POLICY_EXPUNGE_DELETES_ALLOWED_SETTING

    • INDEX_MERGE_POLICY_FLOOR_SEGMENT_SETTING

      public static final Setting<org.opensearch.core.common.unit.ByteSizeValue> INDEX_MERGE_POLICY_FLOOR_SEGMENT_SETTING

    • INDEX_MERGE_POLICY_MAX_MERGE_AT_ONCE_SETTING

      public static final Setting<Integer> INDEX_MERGE_POLICY_MAX_MERGE_AT_ONCE_SETTING

    • INDEX_MERGE_POLICY_MAX_MERGED_SEGMENT_SETTING

      public static final Setting<org.opensearch.core.common.unit.ByteSizeValue> INDEX_MERGE_POLICY_MAX_MERGED_SEGMENT_SETTING

    • INDEX_MERGE_POLICY_SEGMENTS_PER_TIER_SETTING

      public static final Setting<Double> INDEX_MERGE_POLICY_SEGMENTS_PER_TIER_SETTING

    • INDEX_MERGE_POLICY_RECLAIM_DELETES_WEIGHT_SETTING

      public static final Setting<Double> INDEX_MERGE_POLICY_RECLAIM_DELETES_WEIGHT_SETTING

    • INDEX_MERGE_POLICY_DELETES_PCT_ALLOWED_SETTING

      public static final Setting<Double> INDEX_MERGE_POLICY_DELETES_PCT_ALLOWED_SETTING

  • Method Details

    • getMergePolicy

      public org.apache.lucene.index.MergePolicy getMergePolicy()
      Description copied from interface: MergePolicyProvider
      Gets the merge policy to be used for index.
      Specified by:
      getMergePolicy in interface MergePolicyProvider
      Returns:
      The merge policy instance.

    • parseNoCFSRatio

      public static double parseNoCFSRatio(String noCFSRatio)

    • toString

      public String toString()
      Overrides:
      toString in class Object