Package org.elasticsearch.test

Class ESIntegTestCase

java.lang.Object
org.junit.Assert
org.apache.lucene.util.LuceneTestCase
org.elasticsearch.test.ESTestCase
org.elasticsearch.test.ESIntegTestCase
Direct Known Subclasses:
AbstractNumericTestCase, AbstractSnapshotIntegTestCase, AbstractTermsTestCase, DiskUsageIntegTestCase, ESBlobStoreRepositoryIntegTestCase, GeoBoundingBoxQueryIntegTestCase, GeoShapeIntegTestCase
@SuppressFileSystems("ExtrasFS") public abstract class ESIntegTestCase extends ESTestCase
ESIntegTestCase is an abstract base class to run integration tests against a JVM private Elasticsearch Cluster. The test class supports 2 different cluster scopes.

The most common test scope is ESIntegTestCase.Scope.SUITE which shares a cluster per test suite.

If the test methods need specific node settings or change persistent and/or transient cluster settings ESIntegTestCase.Scope.TEST should be used. To configure a scope for the test cluster the ESIntegTestCase.ClusterScope annotation should be used, here is an example: 


 @NodeScope(scope=Scope.TEST) public class SomeIT extends ESIntegTestCase {
 public void testMethod() {}
 }

If no ESIntegTestCase.ClusterScope annotation is present on an integration test the default scope is ESIntegTestCase.Scope.SUITE

A test cluster creates a set of nodes in the background before the test starts. The number of nodes in the cluster is determined at random and can change across tests. The ESIntegTestCase.ClusterScope allows configuring the initial number of nodes that are created before the tests start. 

 @NodeScope(scope=Scope.SUITE, numDataNodes=3)
 public class SomeIT extends ESIntegTestCase {
 public void testMethod() {}
 }

Note, the ESIntegTestCase uses randomized settings on a cluster and index level. For instance each test might use different directory implementation for each test or will return a random client to one of the nodes in the cluster for each call to client(). Test failures might only be reproducible if the correct system properties are passed to the test execution environment.

This class supports the following system properties (passed with -Dkey=value to the application)

  • -D"tests.client.ratio" - a double value in the interval [0..1] which defines the ration between node and transport clients used
  • -D"tests.enable_mock_modules" - a boolean value to enable or disable mock modules. This is useful to test the system without asserting modules that to make sure they don't hide any bugs in production.
  • - a random seed used to initialize the index random context.

  • Field Details

    • SYSPROP_THIRDPARTY

      public static final String SYSPROP_THIRDPARTY
      Property that controls whether ThirdParty Integration tests are run (not the default).
      See Also:

    • SUITE_CLUSTER_NODE_PREFIX

      public static final String SUITE_CLUSTER_NODE_PREFIX
      node names of the corresponding clusters will start with these prefixes
      See Also:

    • TEST_CLUSTER_NODE_PREFIX

      public static final String TEST_CLUSTER_NODE_PREFIX
      See Also:

    • TESTS_CLIENT_RATIO

      public static final String TESTS_CLIENT_RATIO
      Key used to set the transport client ratio via the commandline -D"tests.client.ratio"
      See Also:

    • TESTS_CLUSTER

      public static final String TESTS_CLUSTER
      Key used to eventually switch to using an external cluster and provide its transport addresses
      See Also:

    • INDEX_TEST_SEED_SETTING

      public static final Setting<Long> INDEX_TEST_SEED_SETTING
      Key used to retrieve the index random seed from the index settings on a running node. The value of this seed can be used to initialize a random context for a specific index. It's set once per test via a generic index template.

    • TESTS_ENABLE_MOCK_MODULES

      public static final String TESTS_ENABLE_MOCK_MODULES
      A boolean value to enable or disable mock modules. This is useful to test the system without asserting modules that to make sure they don't hide any bugs in production.
      See Also:

    • DEFAULT_MIN_NUM_SHARDS

      protected static final int DEFAULT_MIN_NUM_SHARDS
      Default minimum number of shards for an index
      See Also:

    • DEFAULT_MAX_NUM_SHARDS

      protected static final int DEFAULT_MAX_NUM_SHARDS
      Default maximum number of shards for an index
      See Also:

  • Constructor Details

    • ESIntegTestCase

      public ESIntegTestCase()

  • Method Details

    • beforeClass

      public static void beforeClass() throws Exception
      Throws:
      Exception

    • enableWarningsCheck

      protected final boolean enableWarningsCheck()
      Description copied from class: ESTestCase
      Whether or not we check after each test whether it has left warnings behind. That happens if any deprecated feature or syntax was used by the test and the test didn't assert on it using ESTestCase.assertWarnings(String...).
      Overrides:
      enableWarningsCheck in class ESTestCase

    • beforeInternal

      protected final void beforeInternal() throws Exception
      Throws:
      Exception

    • setRandomIndexSettings

      protected Settings.Builder setRandomIndexSettings(Random random, Settings.Builder builder)

    • excludeTemplates

      protected Set<String> excludeTemplates()
      Returns:
      An exclude set of index templates that will not be removed in between tests.

    • beforeIndexDeletion

      protected void beforeIndexDeletion() throws Exception
      Throws:
      Exception

    • cluster

      public static TestCluster cluster()

    • isInternalCluster

      public static boolean isInternalCluster()

    • internalCluster

      public static InternalTestCluster internalCluster()

    • clusterService

      public ClusterService clusterService()

    • client

      public static Client client()

    • client

      public static Client client(@Nullable String node)

    • dataNodeClient

      public static Client dataNodeClient()

    • clients

      public static Iterable<Client> clients()

    • minimumNumberOfShards

      protected int minimumNumberOfShards()

    • maximumNumberOfShards

      protected int maximumNumberOfShards()

    • numberOfShards

      protected int numberOfShards()

    • minimumNumberOfReplicas

      protected int minimumNumberOfReplicas()

    • maximumNumberOfReplicas

      protected int maximumNumberOfReplicas()

    • numberOfReplicas

      protected int numberOfReplicas()

    • setDisruptionScheme

      public void setDisruptionScheme(ServiceDisruptionScheme scheme)

    • isolateMasterDisruption

      protected static NetworkDisruption isolateMasterDisruption(NetworkDisruption.NetworkLinkDisruptionType disruptionType)
      Creates a disruption that isolates the current master node from all other nodes in the cluster.
      Parameters:
      disruptionType - type of disruption to create
      Returns:
      disruption

    • indexSettings

      public Settings indexSettings()
      Returns a settings object used in createIndex(String...) and prepareCreate(String) and friends. This method can be overwritten by subclasses to set defaults for the indices that are created by the test. By default it returns a settings object that sets a random number of shards. Number of shards and replicas can be controlled through specific methods.

    • createIndex

      public final void createIndex(String... names)
      Creates one or more indices and asserts that the indices are acknowledged. If one of the indices already exists this method will fail and wipe all the indices created so far.

    • createIndex

      public final void createIndex(String name, Settings indexSettings)
      creates an index with the given setting

    • prepareCreate

      public final CreateIndexRequestBuilder prepareCreate(String index)
      Creates a new CreateIndexRequestBuilder with the settings obtained from indexSettings().

    • prepareCreate

      public final CreateIndexRequestBuilder prepareCreate(String index, int numNodes)
      Creates a new CreateIndexRequestBuilder with the settings obtained from indexSettings(). The index that is created with this builder will only be allowed to allocate on the number of nodes passed to this method.

      This method uses allocation deciders to filter out certain nodes to allocate the created index on. It defines allocation rules based on index.routing.allocation.exclude._name.

    • prepareCreate

      public CreateIndexRequestBuilder prepareCreate(String index, Settings.Builder settingsBuilder)
      Creates a new CreateIndexRequestBuilder with the settings obtained from indexSettings(), augmented by the given builder

    • prepareCreate

      public CreateIndexRequestBuilder prepareCreate(String index, int numNodes, Settings.Builder settingsBuilder)
      Creates a new CreateIndexRequestBuilder with the settings obtained from indexSettings(). The index that is created with this builder will only be allowed to allocate on the number of nodes passed to this method.

      This method uses allocation deciders to filter out certain nodes to allocate the created index on. It defines allocation rules based on index.routing.allocation.exclude._name.

    • waitNoPendingTasksOnAll

      public void waitNoPendingTasksOnAll() throws Exception
      Waits until all nodes have no pending tasks.
      Throws:
      Exception

    • assertResultsAndLogOnFailure

      public void assertResultsAndLogOnFailure(long expectedResults, SearchResponse searchResponse)
      Ensures the result counts are as expected, and logs the results if different

    • allowNodes

      public void allowNodes(String index, int n)
      Restricts the given index to be allocated on n nodes using the allocation deciders. Yet if the shards can't be allocated on any other node shards for this index will remain allocated on more than n nodes.

    • ensureGreen

      public ClusterHealthStatus ensureGreen(String... indices)
      Ensures the cluster has a green state via the cluster health API. This method will also wait for relocations. It is useful to ensure that all action on the cluster have finished and all shards that were currently relocating are now allocated and started.

    • ensureGreen

      public ClusterHealthStatus ensureGreen(org.elasticsearch.core.TimeValue timeout, String... indices)
      Ensures the cluster has a green state via the cluster health API. This method will also wait for relocations. It is useful to ensure that all action on the cluster have finished and all shards that were currently relocating are now allocated and started.
      Parameters:
      timeout - time out value to set on ClusterHealthRequest

    • ensureYellow

      public ClusterHealthStatus ensureYellow(String... indices)
      Ensures the cluster has a yellow state via the cluster health API.

    • ensureYellowAndNoInitializingShards

      public ClusterHealthStatus ensureYellowAndNoInitializingShards(String... indices)
      Ensures the cluster has a yellow state via the cluster health API and ensures the that cluster has no initializing shards for the given indices

    • waitForRelocation

      public ClusterHealthStatus waitForRelocation()
      Waits for all relocating shards to become active using the cluster health API.

    • waitForRelocation

      public ClusterHealthStatus waitForRelocation(ClusterHealthStatus status)
      Waits for all relocating shards to become active and the cluster has reached the given health status using the cluster health API.

    • waitForDocs

      public void waitForDocs(long numDocs, BackgroundIndexer indexer) throws Exception
      Waits until at least a give number of document is visible for searchers
      Parameters:
      numDocs - number of documents to wait for
      indexer - a BackgroundIndexer. It will be first checked for documents indexed. This saves on unneeded searches.
      Throws:
      Exception

    • logClusterState

      public void logClusterState()
      Prints the current cluster state as debug logging.

    • ensureClusterSizeConsistency

      protected void ensureClusterSizeConsistency()

    • ensureClusterStateConsistency

      protected void ensureClusterStateConsistency() throws IOException
      Verifies that all nodes that have the same version of the cluster state as master have same cluster state
      Throws:
      IOException

    • ensureClusterStateCanBeReadByNodeTool

      protected void ensureClusterStateCanBeReadByNodeTool() throws IOException
      Throws:
      IOException

    • refreshClusterInfo

      public static void refreshClusterInfo()

    • ensureSearchable

      protected ClusterHealthStatus ensureSearchable(String... indices)
      Ensures the cluster is in a searchable state for the given indices. This means a searchable copy of each shard is available on the cluster.

    • ensureStableCluster

      protected void ensureStableCluster(int nodeCount)

    • ensureStableCluster

      protected void ensureStableCluster(int nodeCount, org.elasticsearch.core.TimeValue timeValue)

    • ensureStableCluster

      protected void ensureStableCluster(int nodeCount, @Nullable String viaNode)

    • ensureStableCluster

      protected void ensureStableCluster(int nodeCount, org.elasticsearch.core.TimeValue timeValue, boolean local, @Nullable String viaNode)

    • ensureFullyConnectedCluster

      protected void ensureFullyConnectedCluster()
      Ensures that all nodes in the cluster are connected to each other. Some network disruptions may leave nodes that are not the master disconnected from each other. NodeConnectionsService will eventually reconnect but it's handy to be able to ensure this happens faster

    • index

      protected final IndexResponse index(String index, String type, org.elasticsearch.xcontent.XContentBuilder source)
      Syntactic sugar for: 
         client().prepareIndex(index, type).setSource(source).execute().actionGet();

    • index

      protected final IndexResponse index(String index, String type, String id, Map<String,Object> source)
      Syntactic sugar for: 
         client().prepareIndex(index, type).setSource(source).execute().actionGet();

    • index

      protected final IndexResponse index(String index, String type, String id, org.elasticsearch.xcontent.XContentBuilder source)
      Syntactic sugar for: 
         return client().prepareIndex(index, type, id).setSource(source).execute().actionGet();

    • index

      protected final IndexResponse index(String index, String type, String id, Object... source)
      Syntactic sugar for: 
         return client().prepareIndex(index, type, id).setSource(source).execute().actionGet();

    • indexDoc

      protected final IndexResponse indexDoc(String index, String id, Object... source)
      Syntactic sugar for: 
         return client().prepareIndex(index).setId(id).setSource(source).execute().actionGet();

    • index

      protected final IndexResponse index(String index, String type, String id, String source)
      Syntactic sugar for: 
         return client().prepareIndex(index, type, id).setSource(source).execute().actionGet();

      where source is a JSON String.

    • refresh

      protected final RefreshResponse refresh(String... indices)
      Waits for relocations and refreshes all indices in the cluster.
      See Also:

    • flushAndRefresh

      protected final void flushAndRefresh(String... indices)
      Flushes and refreshes all indices in the cluster

    • flush

      protected final FlushResponse flush(String... indices)
      Flush some or all indices in the cluster.

    • forceMerge

      protected ForceMergeResponse forceMerge()
      Waits for all relocations and force merge all indices in the cluster to 1 segment.

    • indexExists

      protected static boolean indexExists(String index)
      Returns true iff the given index exists otherwise false

    • enableAllocation

      protected final void enableAllocation(String... indices)
      Syntactic sugar for enabling allocation for indices

    • disableAllocation

      protected final void disableAllocation(String... indices)
      Syntactic sugar for disabling allocation for indices

    • admin

      protected static AdminClient admin()
      Returns a random admin client. This client can either be a node or a transport client pointing to any of the nodes in the cluster.

    • clusterAdmin

      protected static ClusterAdminClient clusterAdmin()
      Returns a random cluster admin client. This client can be pointing to any of the nodes in the cluster.

    • indexRandom

      public void indexRandom(boolean forceRefresh, IndexRequestBuilder... builders) throws InterruptedException
      Convenience method that forwards to indexRandom(boolean, List).
      Throws:
      InterruptedException

    • indexRandom

      public void indexRandom(boolean forceRefresh, boolean dummyDocuments, IndexRequestBuilder... builders) throws InterruptedException
      Throws:
      InterruptedException

    • indexRandom

      public void indexRandom(boolean forceRefresh, List<IndexRequestBuilder> builders) throws InterruptedException
      Indexes the given IndexRequestBuilder instances randomly. It shuffles the given builders and either indexes them in a blocking or async fashion. This is very useful to catch problems that relate to internal document ids or index segment creations. Some features might have bug when a given document is the first or the last in a segment or if only one document is in a segment etc. This method prevents issues like this by randomizing the index layout.
      Parameters:
      forceRefresh - if true all involved indices are refreshed once the documents are indexed. Additionally if true some empty dummy documents are may be randomly inserted into the document list and deleted once all documents are indexed. This is useful to produce deleted documents on the server side.
      builders - the documents to index.
      Throws:
      InterruptedException
      See Also:

    • indexRandom

      public void indexRandom(boolean forceRefresh, boolean dummyDocuments, List<IndexRequestBuilder> builders) throws InterruptedException
      Indexes the given IndexRequestBuilder instances randomly. It shuffles the given builders and either indexes them in a blocking or async fashion. This is very useful to catch problems that relate to internal document ids or index segment creations. Some features might have bug when a given document is the first or the last in a segment or if only one document is in a segment etc. This method prevents issues like this by randomizing the index layout.
      Parameters:
      forceRefresh - if true all involved indices are refreshed once the documents are indexed.
      dummyDocuments - if true some empty dummy documents may be randomly inserted into the document list and deleted once all documents are indexed. This is useful to produce deleted documents on the server side.
      builders - the documents to index.
      Throws:
      InterruptedException

    • indexRandom

      public void indexRandom(boolean forceRefresh, boolean dummyDocuments, boolean maybeFlush, List<IndexRequestBuilder> builders) throws InterruptedException
      Indexes the given IndexRequestBuilder instances randomly. It shuffles the given builders and either indexes them in a blocking or async fashion. This is very useful to catch problems that relate to internal document ids or index segment creations. Some features might have bug when a given document is the first or the last in a segment or if only one document is in a segment etc. This method prevents issues like this by randomizing the index layout.
      Parameters:
      forceRefresh - if true all involved indices are refreshed once the documents are indexed.
      dummyDocuments - if true some empty dummy documents may be randomly inserted into the document list and deleted once all documents are indexed. This is useful to produce deleted documents on the server side.
      maybeFlush - if true this method may randomly execute full flushes after index operations.
      builders - the documents to index.
      Throws:
      InterruptedException

    • disableIndexBlock

      public static void disableIndexBlock(String index, String block)
      Disables an index block for the specified index

    • enableIndexBlock

      public static void enableIndexBlock(String index, String block)
      Enables an index block for the specified index

    • setClusterReadOnly

      public static void setClusterReadOnly(boolean value)
      Sets or unsets the cluster read_only mode

    • clearScroll

      public void clearScroll(String... scrollIds)
      Clears the given scroll Ids

    • nodeSettings

      protected Settings nodeSettings(int nodeOrdinal, Settings otherSettings)
      This method is used to obtain settings for the Nth node in the cluster. Nodes in this cluster are associated with an ordinal number such that nodes can be started with specific configurations. This method might be called multiple times with the same ordinal and is expected to return the same value for each invocation. In other words subclasses must ensure this method is idempotent.

    • nodeConfigPath

      protected Path nodeConfigPath(int nodeOrdinal)

    • nodePlugins

      protected Collection<Class<? extends Plugin>> nodePlugins()
      Returns a collection of plugins that should be loaded on each node.

    • transportClientPlugins

      protected Collection<Class<? extends Plugin>> transportClientPlugins()
      Returns a collection of plugins that should be loaded when creating a transport client.

    • transportClientSettings

      protected Settings transportClientSettings()
      This method is used to obtain additional settings for clients created by the internal cluster. These settings will be applied on the client in addition to some randomized settings defined in the cluster. These settings will also override any other settings the internal cluster might add by default.

    • externalClusterClientSettings

      protected Settings externalClusterClientSettings()

    • ignoreExternalCluster

      protected boolean ignoreExternalCluster()

    • buildTestCluster

      protected TestCluster buildTestCluster(ESIntegTestCase.Scope scope, long seed) throws IOException
      Throws:
      IOException

    • addMockTransportService

      protected boolean addMockTransportService()
      Iff this returns true mock transport implementations are used for the test runs. Otherwise not mock transport impls are used. The default is true.

    • addMockHttpTransport

      protected boolean addMockHttpTransport()
      Returns true iff this test cluster should use a dummy http transport

    • addMockInternalEngine

      protected boolean addMockInternalEngine()
      Returns true if this test cluster can use a mock internal engine. Defaults to true.

    • addMockGeoShapeFieldMapper

      protected boolean addMockGeoShapeFieldMapper()
      Returns true iff this test cluster should use a dummy geo_shape field mapper

    • getClientWrapper

      protected Function<Client,Client> getClientWrapper()
      Returns a function that allows to wrap / filter all clients that are exposed by the test cluster. This is useful for debugging or request / response pre and post processing. It also allows to intercept all calls done by the test framework. By default this method returns an identity function Function.identity().

    • getMockPlugins

      protected Collection<Class<? extends Plugin>> getMockPlugins()
      Return the mock plugins the cluster should use

    • getPerTestTransportClientRatio

      protected double getPerTestTransportClientRatio()
      Returns the transport client ratio from the class level annotation or via System.getProperty(String) if available. If both are not available this will return a random ratio in the interval [0..1].

    • randomRepoPath

      public static Path randomRepoPath()
      Returns path to a random directory that can be used to create a temporary file system repo

    • randomRepoPath

      public static Path randomRepoPath(Settings settings)
      Returns path to a random directory that can be used to create a temporary file system repo

    • getNumShards

      protected ESIntegTestCase.NumShards getNumShards(String index)

    • assertAllShardsOnNodes

      public Set<String> assertAllShardsOnNodes(String index, String... pattern)
      Asserts that all shards are allocated on nodes matching the given node pattern.

    • assertSortedSegments

      public void assertSortedSegments(String indexName, org.apache.lucene.search.Sort expectedIndexSort)
      Asserts that all segments are sorted with the provided Sort.

    • setupTestCluster

      public final void setupTestCluster() throws Exception
      Throws:
      Exception

    • cleanUpCluster

      public final void cleanUpCluster() throws Exception
      Throws:
      Exception

    • afterClass

      public static void afterClass() throws Exception
      Throws:
      Exception

    • routingKeyForShard

      protected String routingKeyForShard(String index, int shard)
      Compute a routing key that will route documents to the shard-th shard of the provided index.

    • xContentRegistry

      protected org.elasticsearch.xcontent.NamedXContentRegistry xContentRegistry()
      Description copied from class: ESTestCase
      The NamedXContentRegistry to use for this test. Subclasses should override and use liberally.
      Overrides:
      xContentRegistry in class ESTestCase

    • forbidPrivateIndexSettings

      protected boolean forbidPrivateIndexSettings()

    • forceSingleDataPath

      protected boolean forceSingleDataPath()
      Override to return true in tests that cannot handle multiple data paths.

    • getRestClient

      protected static RestClient getRestClient()
      Returns an instance of RestClient pointing to the current test cluster. Creates a new client if the method is invoked for the first time in the context of the current test scope. The returned client gets automatically closed when needed, it shouldn't be closed as part of tests otherwise it cannot be reused by other tests anymore.

    • createRestClient

      protected static RestClient createRestClient()

    • createRestClient

      protected static RestClient createRestClient(RestClientBuilder.HttpClientConfigCallback httpClientConfigCallback, String protocol)

    • createRestClient

      protected static RestClient createRestClient(List<NodeInfo> nodes, RestClientBuilder.HttpClientConfigCallback httpClientConfigCallback, String protocol)

    • setupSuiteScopeCluster

      protected void setupSuiteScopeCluster() throws Exception
      This method is executed iff the test is annotated with ESIntegTestCase.SuiteScopeTestCase before the first test of this class is executed.
      Throws:
      Exception
      See Also:

    • resolveIndex

      public static Index resolveIndex(String index)

    • resolveCustomDataPath

      public static String resolveCustomDataPath(String index)

    • inFipsJvm

      public static boolean inFipsJvm()

    • restartNodesOnBrokenClusterState

      protected void restartNodesOnBrokenClusterState(ClusterState.Builder clusterStateBuilder) throws Exception
      Throws:
      Exception