Various Elasticsearch features such as Security or Watcher keep their state in their own indices. We keep these indices separate from the user index space for a few reasons. In some cases, the indices contain information that should be hidden from users. But, more generally, we want to protect these indices and data streams from being inadvertently modified or deleted.
The system resources are grouped by feature, using the SystemIndices.Feature
class. Most features will be loaded from
instances of SystemIndexPlugin
; any other features will be described in this class. Features may be retrieved by name or
iterated over (see getFeature(String)
and getFeatures()
). Each Feature provides collections of
SystemIndexDescriptor
s or SystemDataStreamDescriptor
s. These descriptors define their resources by means of patterns.
Any index name or data stream name that matches one of these patterns is considered a system resource for * that descriptor, and user
access to it will be restricted. These patterns are gathered and validated so that the SystemIndices object can provide information
about system resources: for example, whether a particular string will be considered a “system name” (see isSystemIndex(String)
).
For more information about the expected behavior of system indices, see SystemIndexDescriptor
. For more information about
the expected behavior of system data streams, see SystemDataStreamDescriptor
.
The SystemIndices object is constructed during Node
startup, and is not modified after construction.
In other words, the set of system resources will be consistent over the lifetime of a node.
System resources will specify thread pools for reads, writes, and searches. This can ensure that system-critical operations, such
as user authentication, are not blocked by heavy thread contention from user activities. getExecutorSelector()
provides an
object for convenient look-ups of these thread pools.
There are a few requirements for system features, collectively:
- The feature names must be distinct.
- System index patterns must not overlap.
- Aliases for system indices must be distinct.
- Feature names must not be reserved names. Right now, the only reserved name is “none”.
System index access is currently controlled by Security role index permissions. However, we have deprecated general rest access to
system indices. This class provides checks for system index “access levels” (see getSystemIndexAccessLevel(ThreadContext)
).
If a request has the wrong access level for a system index it is targeting, then we will issue a deprecation warning. In the future,
we will block access. The non-deprecated way to access certain external system indices is to use the correct request headers. This
behavior is already in place in SystemDataStreamDescriptor
and “net-new” system indices (see
SystemIndexDescriptor.isNetNew()
).
The implementation of the system index name checks makes heavy use of the Lucene Automaton
class. At a high level, an
automaton is a kind of matcher that can be created from a regex. Lucene Automata give us the ability to check for overlapping
patterns, and to create efficient unions of patterns.
-
Nested Class Summary
Modifier and TypeClassDescriptionstatic class
Describes an Elasticsearch system feature that keeps state in protected indices and data streams.static enum
In a future release, these access levels will be used to allow or deny requests for system resources. -
Field Summary
-
Constructor Summary
ConstructorDescriptionSystemIndices
(List<SystemIndices.Feature> pluginAndModuleFeatures) Initialize the SystemIndices object -
Method Summary
Modifier and TypeMethodDescriptionstatic IllegalArgumentException
dataStreamAccessException
(ThreadContext threadContext, Collection<String> names) Finds a single matchingSystemDataStreamDescriptor
, if any, for the given DataStream name.findMatchingDescriptor
(String name) Finds a single matchingSystemIndexDescriptor
, if any, for the given index name.Used to determine which executor should be used for operations on this index.getFeature
(String name) Get a feature by name.Get a set of feature names.Get a collection of the Features this SystemIndices object is managing.getProductSystemIndexNamePredicate
(ThreadContext threadContext) Builds a predicate that tests whether a system index should be accessible for a given ThreadContext.getSystemIndexAccessLevel
(ThreadContext threadContext) Determines what level of system index access should be allowed in the current context.org.apache.lucene.util.automaton.Automaton
The Elasticsearch security plugin can use the automaton that matches all system resource names to efficiently authorize requests.boolean
isNetNewSystemIndex
(String indexName) Checks whether an index is a net-new system index, meaning we can apply non-BWC behavior to it.boolean
isSystemDataStream
(String name) Determines whether the provided name matches that of a system data stream that has been defined by aSystemDataStreamDescriptor
boolean
isSystemIndex
(String indexName) Determines whether a given index is a system index by comparing its name to the collection of loadedSystemIndexDescriptor
s.boolean
isSystemIndex
(Index index) Determines whether a given index is a system index by comparing its name to the collection of loadedSystemIndexDescriptor
sboolean
Determines whether the provided name matches that of an index that backs a system data stream.boolean
isSystemName
(String name) Checks whether the given name matches a reserved name or pattern that is intended for use by a system component.static IllegalArgumentException
netNewSystemIndexAccessException
(ThreadContext threadContext, Collection<String> names) validateDataStreamAccess
(String dataStreamName, ThreadContext threadContext) static void
validateFeatureName
(String name, String plugin) Check that a feature name is not reserved
-
Field Details
-
SYSTEM_INDEX_ACCESS_CONTROL_HEADER_KEY
- See Also:
-
EXTERNAL_SYSTEM_INDEX_ACCESS_CONTROL_HEADER_KEY
- See Also:
-
UPGRADED_INDEX_SUFFIX
- See Also:
-
SERVER_SYSTEM_MAPPINGS_VERSIONS
public static final Map<String,SystemIndexDescriptor.MappingsVersion> SERVER_SYSTEM_MAPPINGS_VERSIONS
-
-
Constructor Details
-
SystemIndices
Initialize the SystemIndices object- Parameters:
pluginAndModuleFeatures
- A list of features from which we will load system indices. These features come from plugins and modules. Non-plugin system features such as Tasks will be added automatically.
-
-
Method Details
-
isSystemName
Checks whether the given name matches a reserved name or pattern that is intended for use by a system component. The name is checked against index names, aliases, data stream names, and the names of indices that back a system data stream. -
isSystemIndex
Determines whether a given index is a system index by comparing its name to the collection of loadedSystemIndexDescriptor
s- Parameters:
index
- theIndex
object to check against loadedSystemIndexDescriptor
s- Returns:
- true if the
Index
's name matches a pattern from aSystemIndexDescriptor
-
isSystemIndex
Determines whether a given index is a system index by comparing its name to the collection of loadedSystemIndexDescriptor
s. This will also match alias names that belong to system indices.- Parameters:
indexName
- the index name to check against loadedSystemIndexDescriptor
s- Returns:
- true if the index name matches a pattern from a
SystemIndexDescriptor
-
isSystemDataStream
Determines whether the provided name matches that of a system data stream that has been defined by aSystemDataStreamDescriptor
-
isSystemIndexBackingDataStream
Determines whether the provided name matches that of an index that backs a system data stream. Backing indices for system data streams are marked as "system" in their metadata (seeSystemIndexMetadataUpgradeService
) and receive the same protections as the system data stream. -
getSystemNameAutomaton
public org.apache.lucene.util.automaton.Automaton getSystemNameAutomaton()The Elasticsearch security plugin can use the automaton that matches all system resource names to efficiently authorize requests.- Returns:
- An
Automaton
that tests whether strings are names of system indices, aliases, or data streams.
-
isNetNewSystemIndex
Checks whether an index is a net-new system index, meaning we can apply non-BWC behavior to it. SeeSystemIndexDescriptor.isNetNew()
.- Parameters:
indexName
- The index name to check.- Returns:
true
if the given index is covered by a net-new system index descriptor,false
otherwise.
-
getExecutorSelector
Used to determine which executor should be used for operations on this index. SeeExecutorSelector
docs for details. -
findMatchingDescriptor
Finds a single matchingSystemIndexDescriptor
, if any, for the given index name.- Parameters:
name
- the name of the index- Returns:
- The matching
SystemIndexDescriptor
ornull
if no descriptor is found
-
findMatchingDataStreamDescriptor
Finds a single matchingSystemDataStreamDescriptor
, if any, for the given DataStream name.- Parameters:
name
- the name of the DataStream- Returns:
- The matching
SystemDataStreamDescriptor
ornull
if no descriptor is found
-
getProductSystemIndexNamePredicate
Builds a predicate that tests whether a system index should be accessible for a given ThreadContext. We allow guaranteed (that is, non-deprecated) external access to system indices based on special request headers in addition to security roles. If those headers are present and provide a product name with access to the index, they will be added to the thread context and checked here. Without these headers, we will add deprecation warnings to the response. In future versions, we will deny access altogether.- Parameters:
threadContext
- the threadContext containing headers used for system index access- Returns:
- Predicate to check external system index names with
-
getFeatureNames
Get a set of feature names. This is useful for checking whether particular features are present on the node.- Returns:
- A set of all feature names
-
getFeature
Get a feature by name.- Parameters:
name
- Name of a feature.- Returns:
- The corresponding feature if it exists on this node, null otherwise.
-
getFeatures
Get a collection of the Features this SystemIndices object is managing.- Returns:
- A collection of Features.
-
validateDataStreamAccess
public SystemDataStreamDescriptor validateDataStreamAccess(String dataStreamName, ThreadContext threadContext) -
dataStreamAccessException
public static IllegalArgumentException dataStreamAccessException(ThreadContext threadContext, Collection<String> names) -
netNewSystemIndexAccessException
public static IllegalArgumentException netNewSystemIndexAccessException(ThreadContext threadContext, Collection<String> names) -
getSystemIndexAccessLevel
public static SystemIndices.SystemIndexAccessLevel getSystemIndexAccessLevel(ThreadContext threadContext) Determines what level of system index access should be allowed in the current context. For system data streams and "net-new" system indices (seeSystemIndexDescriptor.isNetNew()
), access levels should be used to reject requests entirely. For non-net-new, backwards-compatible system indices, these access levels should be used for deprecation warnings.- Parameters:
threadContext
- the current thread context that has headers associated with the current request- Returns:
SystemIndices.SystemIndexAccessLevel.ALL
if unrestricted system index access should be allowed,SystemIndices.SystemIndexAccessLevel.RESTRICTED
if a subset of system index access should be allowed, orSystemIndices.SystemIndexAccessLevel.NONE
if no system index access should be allowed.
-
getMappingsVersions
-
validateFeatureName
Check that a feature name is not reserved- Parameters:
name
- Name of featureplugin
- Name of plugin providing the feature
-