Package org.elasticsearch.index.shard

Class IndexShardTestCase

java.lang.Object
org.junit.Assert
org.apache.lucene.tests.util.LuceneTestCase
org.elasticsearch.test.ESTestCase
org.elasticsearch.index.shard.IndexShardTestCase
Direct Known Subclasses:
ESIndexLevelReplicationTestCase
public abstract class IndexShardTestCase extends ESTestCase
A base class for unit tests that need to create and shutdown IndexShard instances easily, containing utilities for shard creation and recoveries. See {newShard(boolean)} and newStartedShard() for a good starting points

  • Field Details

  • Constructor Details

    • IndexShardTestCase

      public IndexShardTestCase()

  • Method Details

    • setUp

      public void setUp() throws Exception
      Overrides:
      setUp in class org.apache.lucene.tests.util.LuceneTestCase
      Throws:
      Exception

    • setUpThreadPool

      protected ThreadPool setUpThreadPool()

    • tearDown

      public void tearDown() throws Exception
      Overrides:
      tearDown in class org.apache.lucene.tests.util.LuceneTestCase
      Throws:
      Exception

    • tearDownThreadPool

      protected void tearDownThreadPool()

    • allowShardFailures

      protected void allowShardFailures()
      by default, tests will fail if any shard created by this class fails. Tests that cause failures by design can call this method to ignore those failures

    • failOnShardFailures

      protected void failOnShardFailures()

    • threadPoolSettings

      public Settings threadPoolSettings()

    • createStore

      protected Store createStore(IndexSettings indexSettings, ShardPath shardPath) throws IOException
      Throws:
      IOException

    • createStore

      protected Store createStore(ShardId shardId, IndexSettings indexSettings, org.apache.lucene.store.Directory directory) throws IOException
      Throws:
      IOException

    • newShard

      protected IndexShard newShard(boolean primary) throws IOException
      Creates a new initializing shard. The shard will have its own unique data path.
      Parameters:
      primary - indicates whether to a primary shard (ready to recover from an empty store) or a replica (ready to recover from another shard)
      Throws:
      IOException

    • newShard

      protected IndexShard newShard(boolean primary, Settings settings) throws IOException
      Creates a new initializing shard. The shard will have its own unique data path.
      Parameters:
      primary - indicates whether to a primary shard (ready to recover from an empty store) or a replica (ready to recover from another shard)
      Throws:
      IOException

    • newShard

      protected IndexShard newShard(boolean primary, Settings settings, EngineFactory engineFactory, IndexingOperationListener... listeners) throws IOException
      Creates a new initializing shard. The shard will have its own unique data path.
      Parameters:
      primary - indicates whether to a primary shard (ready to recover from an empty store) or a replica (ready to recover from another shard)
      settings - the settings to use for this shard
      engineFactory - the engine factory to use for this shard
      listeners - the indexing operation listeners to add
      Throws:
      IOException

    • newShard

      protected IndexShard newShard(ShardRouting shardRouting, IndexingOperationListener... listeners) throws IOException
      Throws:
      IOException

    • newShard

      protected IndexShard newShard(ShardRouting shardRouting, Settings settings, IndexingOperationListener... listeners) throws IOException
      Throws:
      IOException

    • newShard

      protected IndexShard newShard(ShardRouting shardRouting, Settings settings, EngineFactory engineFactory, IndexingOperationListener... listeners) throws IOException
      Creates a new initializing shard. The shard will have its own unique data path.
      Parameters:
      shardRouting - the ShardRouting to use for this shard
      settings - the settings to use for this shard
      engineFactory - the engine factory to use for this shard
      listeners - an optional set of listeners to add to the shard
      Throws:
      IOException

    • newShard

      protected IndexShard newShard(ShardId shardId, boolean primary, IndexingOperationListener... listeners) throws IOException
      creates a new initializing shard. The shard will have its own unique data path.
      Parameters:
      shardId - the shard id to use
      primary - indicates whether to a primary shard (ready to recover from an empty store) or a replica (ready to recover from another shard)
      listeners - an optional set of listeners to add to the shard
      Throws:
      IOException

    • newShard

      protected IndexShard newShard(ShardId shardId, boolean primary, String nodeId, IndexMetadata indexMetadata, @Nullable org.elasticsearch.core.CheckedFunction<org.apache.lucene.index.DirectoryReader,org.apache.lucene.index.DirectoryReader,IOException> readerWrapper) throws IOException
      creates a new initializing shard. The shard will will be put in its proper path under the supplied node id.
      Parameters:
      shardId - the shard id to use
      primary - indicates whether to a primary shard (ready to recover from an empty store) or a replica (ready to recover from another shard)
      Throws:
      IOException

    • newShard

      protected IndexShard newShard(ShardId shardId, boolean primary, String nodeId, IndexMetadata indexMetadata, @Nullable org.elasticsearch.core.CheckedFunction<org.apache.lucene.index.DirectoryReader,org.apache.lucene.index.DirectoryReader,IOException> readerWrapper, GlobalCheckpointSyncer globalCheckpointSyncer) throws IOException
      creates a new initializing shard. The shard will will be put in its proper path under the supplied node id.
      Parameters:
      shardId - the shard id to use
      primary - indicates whether to a primary shard (ready to recover from an empty store) or a replica (ready to recover from another shard)
      Throws:
      IOException

    • newShard

      protected IndexShard newShard(ShardRouting routing, IndexMetadata indexMetadata, @Nullable org.elasticsearch.core.CheckedFunction<org.apache.lucene.index.DirectoryReader,org.apache.lucene.index.DirectoryReader,IOException> indexReaderWrapper, EngineFactory engineFactory, IndexingOperationListener... listeners) throws IOException
      creates a new initializing shard. The shard will will be put in its proper path under the current node id the shard is assigned to.
      Parameters:
      routing - shard routing to use
      indexMetadata - indexMetadata for the shard, including any mapping
      listeners - an optional set of listeners to add to the shard
      Throws:
      IOException

    • newShard

      protected IndexShard newShard(ShardRouting routing, IndexMetadata indexMetadata, @Nullable org.elasticsearch.core.CheckedFunction<org.apache.lucene.index.DirectoryReader,org.apache.lucene.index.DirectoryReader,IOException> indexReaderWrapper, @Nullable EngineFactory engineFactory, GlobalCheckpointSyncer globalCheckpointSyncer, RetentionLeaseSyncer retentionLeaseSyncer, IndexingOperationListener... listeners) throws IOException
      creates a new initializing shard. The shard will will be put in its proper path under the current node id the shard is assigned to.
      Parameters:
      routing - shard routing to use
      indexMetadata - indexMetadata for the shard, including any mapping
      indexReaderWrapper - an optional wrapper to be used during search
      globalCheckpointSyncer - callback for syncing global checkpoints
      listeners - an optional set of listeners to add to the shard
      Throws:
      IOException

    • newShard

      protected IndexShard newShard(ShardRouting routing, ShardPath shardPath, IndexMetadata indexMetadata, @Nullable org.elasticsearch.core.CheckedFunction<IndexSettings,Store,IOException> storeProvider, @Nullable org.elasticsearch.core.CheckedFunction<org.apache.lucene.index.DirectoryReader,org.apache.lucene.index.DirectoryReader,IOException> indexReaderWrapper, @Nullable EngineFactory engineFactory, GlobalCheckpointSyncer globalCheckpointSyncer, RetentionLeaseSyncer retentionLeaseSyncer, IndexEventListener indexEventListener, IndexingOperationListener... listeners) throws IOException
      creates a new initializing shard.
      Parameters:
      routing - shard routing to use
      shardPath - path to use for shard data
      indexMetadata - indexMetadata for the shard, including any mapping
      storeProvider - an optional custom store provider to use. If null a default file based store will be created
      indexReaderWrapper - an optional wrapper to be used during search
      globalCheckpointSyncer - callback for syncing global checkpoints
      indexEventListener - index event listener
      listeners - an optional set of listeners to add to the shard
      Throws:
      IOException

    • newShard

      protected IndexShard newShard(ShardRouting routing, ShardPath shardPath, IndexMetadata indexMetadata, @Nullable org.elasticsearch.core.CheckedFunction<IndexSettings,Store,IOException> storeProvider, @Nullable org.elasticsearch.core.CheckedFunction<org.apache.lucene.index.DirectoryReader,org.apache.lucene.index.DirectoryReader,IOException> indexReaderWrapper, @Nullable EngineFactory engineFactory, GlobalCheckpointSyncer globalCheckpointSyncer, RetentionLeaseSyncer retentionLeaseSyncer, IndexEventListener indexEventListener, LongSupplier relativeTimeSupplier, IndexingOperationListener... listeners) throws IOException
      creates a new initializing shard.
      Parameters:
      routing - shard routing to use
      shardPath - path to use for shard data
      indexMetadata - indexMetadata for the shard, including any mapping
      storeProvider - an optional custom store provider to use. If null a default file based store will be created
      indexReaderWrapper - an optional wrapper to be used during search
      globalCheckpointSyncer - callback for syncing global checkpoints
      indexEventListener - index event listener
      relativeTimeSupplier - the clock used to measure relative time
      listeners - an optional set of listeners to add to the shard
      Throws:
      IOException

    • reinitShard

      protected IndexShard reinitShard(IndexShard current, IndexingOperationListener... listeners) throws IOException
      Takes an existing shard, closes it and starts a new initialing shard at the same location
      Parameters:
      listeners - new listerns to use for the newly created shard
      Throws:
      IOException

    • reinitShard

      protected IndexShard reinitShard(IndexShard current, ShardRouting routing, IndexingOperationListener... listeners) throws IOException
      Takes an existing shard, closes it and starts a new initialing shard at the same location
      Parameters:
      routing - the shard routing to use for the newly created shard.
      listeners - new listerns to use for the newly created shard
      Throws:
      IOException

    • reinitShard

      protected IndexShard reinitShard(IndexShard current, ShardRouting routing, IndexMetadata indexMetadata, EngineFactory engineFactory, IndexingOperationListener... listeners) throws IOException
      Takes an existing shard, closes it and starts a new initialing shard at the same location
      Parameters:
      routing - the shard routing to use for the newly created shard.
      listeners - new listerns to use for the newly created shard
      indexMetadata - the index metadata to use for the newly created shard
      engineFactory - the engine factory for the new shard
      Throws:
      IOException

    • newStartedShard

      protected IndexShard newStartedShard() throws IOException
      Creates a new empty shard and starts it. The shard will randomly be a replica or a primary.
      Throws:
      IOException

    • newStartedShard

      protected IndexShard newStartedShard(Settings settings) throws IOException
      Creates a new empty shard and starts it
      Parameters:
      settings - the settings to use for this shard
      Throws:
      IOException

    • newStartedShard

      protected IndexShard newStartedShard(boolean primary) throws IOException
      Creates a new empty shard and starts it.
      Parameters:
      primary - controls whether the shard will be a primary or a replica.
      Throws:
      IOException

    • newStartedShard

      protected IndexShard newStartedShard(boolean primary, Settings settings, IndexingOperationListener... listeners) throws IOException
      Creates a new empty shard and starts it.
      Parameters:
      primary - controls whether the shard will be a primary or a replica.
      settings - the settings to use for this shard
      listeners - the indexing operation listeners to add
      Throws:
      IOException

    • newStartedShard

      protected IndexShard newStartedShard(boolean primary, Settings settings, EngineFactory engineFactory, IndexingOperationListener... listeners) throws IOException
      Creates a new empty shard with the specified settings and engine factory and starts it.
      Parameters:
      primary - controls whether the shard will be a primary or a replica.
      settings - the settings to use for this shard
      engineFactory - the engine factory to use for this shard
      listeners - the indexing operation listeners to add
      Throws:
      IOException

    • newStartedShard

      protected IndexShard newStartedShard(org.elasticsearch.core.CheckedFunction<Boolean,IndexShard,IOException> shardFunction, boolean primary) throws IOException
      creates a new empty shard and starts it.
      Parameters:
      shardFunction - shard factory function
      primary - controls whether the shard will be a primary or a replica.
      Throws:
      IOException

    • closeShards

      protected void closeShards(IndexShard... shards) throws IOException
      Throws:
      IOException

    • closeShard

      protected void closeShard(IndexShard shard, boolean assertConsistencyBetweenTranslogAndLucene) throws IOException
      Throws:
      IOException

    • closeShards

      protected void closeShards(Iterable<IndexShard> shards) throws IOException
      Throws:
      IOException

    • recoverShardFromStore

      protected void recoverShardFromStore(IndexShard primary) throws IOException
      Throws:
      IOException

    • updateRoutingEntry

      public static void updateRoutingEntry(IndexShard shard, ShardRouting shardRouting) throws IOException
      Throws:
      IOException

    • recoveryEmptyReplica

      protected void recoveryEmptyReplica(IndexShard replica, boolean startReplica) throws IOException
      Throws:
      IOException

    • getFakeDiscoNode

      protected DiscoveryNode getFakeDiscoNode(String id)

    • recoverReplica

      protected void recoverReplica(IndexShard replica, IndexShard primary, boolean startReplica) throws IOException
      recovers a replica from the given primary
      Throws:
      IOException

    • recoverReplica

      protected void recoverReplica(IndexShard replica, IndexShard primary, BiFunction<IndexShard,DiscoveryNode,RecoveryTarget> targetSupplier, boolean markAsRecovering, boolean markAsStarted) throws IOException
      recovers a replica from the given primary
      Throws:
      IOException

    • recoverUnstartedReplica

      protected final void recoverUnstartedReplica(IndexShard replica, IndexShard primary, BiFunction<IndexShard,DiscoveryNode,RecoveryTarget> targetSupplier, boolean markAsRecovering, Set<String> inSyncIds, IndexShardRoutingTable routingTable) throws IOException
      Recovers a replica from the give primary, allow the user to supply a custom recovery target. A typical usage of a custom recovery target is to assert things in the various stages of recovery. Note: this method keeps the shard in IndexShardState.POST_RECOVERY and doesn't start it.
      Parameters:
      replica - the recovery target shard
      primary - the recovery source shard
      targetSupplier - supplies an instance of RecoveryTarget
      markAsRecovering - set to false if the replica is marked as recovering
      Throws:
      IOException

    • startReplicaAfterRecovery

      protected void startReplicaAfterRecovery(IndexShard replica, IndexShard primary, Set<String> inSyncIds, IndexShardRoutingTable routingTable) throws IOException
      Throws:
      IOException

    • promoteReplica

      protected void promoteReplica(IndexShard replica, Set<String> inSyncIds, IndexShardRoutingTable routingTable) throws IOException
      promotes a replica to primary, incrementing it's term and starting it if needed
      Throws:
      IOException

    • getOperationPermit

      public static org.elasticsearch.core.Releasable getOperationPermit(IndexShard shard)

    • getShardDocUIDs

      public static Set<String> getShardDocUIDs(IndexShard shard) throws IOException
      Throws:
      IOException

    • getDocIdAndSeqNos

      public static List<DocIdSeqNoAndSource> getDocIdAndSeqNos(IndexShard shard) throws IOException
      Throws:
      IOException

    • assertDocCount

      protected void assertDocCount(IndexShard shard, int docDount) throws IOException
      Throws:
      IOException

    • assertDocs

      protected void assertDocs(IndexShard shard, String... ids) throws IOException
      Throws:
      IOException

    • assertConsistentHistoryBetweenTranslogAndLucene

      public static void assertConsistentHistoryBetweenTranslogAndLucene(IndexShard shard) throws IOException
      Throws:
      IOException

    • indexDoc

      protected Engine.IndexResult indexDoc(IndexShard shard, String type, String id) throws IOException
      Throws:
      IOException

    • indexDoc

      protected Engine.IndexResult indexDoc(IndexShard shard, String type, String id, String source) throws IOException
      Throws:
      IOException

    • indexDoc

      protected Engine.IndexResult indexDoc(IndexShard shard, String id, String source, org.elasticsearch.xcontent.XContentType xContentType, String routing) throws IOException
      Throws:
      IOException

    • updateMappings

      protected void updateMappings(IndexShard shard, IndexMetadata indexMetadata)

    • deleteDoc

      protected Engine.DeleteResult deleteDoc(IndexShard shard, String id) throws IOException
      Throws:
      IOException

    • flushShard

      protected void flushShard(IndexShard shard)

    • flushShard

      protected void flushShard(IndexShard shard, boolean force)

    • recoverFromStore

      public static boolean recoverFromStore(IndexShard newShard)

    • recoverShardFromSnapshot

      protected void recoverShardFromSnapshot(IndexShard shard, Snapshot snapshot, Repository repository)
      Recover a shard from a snapshot using a given repository

    • snapshotShard

      protected ShardGeneration snapshotShard(IndexShard shard, Snapshot snapshot, Repository repository) throws IOException
      Snapshot a shard using a given repository.
      Returns:
      new shard generation
      Throws:
      IOException

    • getEngine

      public static Engine getEngine(IndexShard indexShard)
      Helper method to access (package-protected) engine from tests

    • getTranslog

      public static Translog getTranslog(IndexShard shard)

    • getReplicationTracker

      public static ReplicationTracker getReplicationTracker(IndexShard indexShard)

    • createTestWarmer

      public static Engine.Warmer createTestWarmer(IndexSettings indexSettings)