Package org.apache.druid.data.input.impl

Class CloudObjectInputSource

java.lang.Object

org.apache.druid.data.input.AbstractInputSource

org.apache.druid.data.input.impl.CloudObjectInputSource

All Implemented Interfaces:

SplittableInputSource<List<CloudObjectLocation>>, SystemFieldInputSource, InputSource

public abstract class CloudObjectInputSource
extends AbstractInputSource
implements SplittableInputSource<List<CloudObjectLocation>>, SystemFieldInputSource

Field Summary

Fields

Modifier and Type	Field	Description
protected SystemFields	systemFields

Fields inherited from interface org.apache.druid.data.input.InputSource

TYPE_PROPERTY

Fields inherited from interface org.apache.druid.data.input.impl.SplittableInputSource

DEFAULT_SPLIT_HINT_SPEC

Fields inherited from interface org.apache.druid.data.input.impl.systemfield.SystemFieldInputSource

SYSTEM_FIELDS_PROPERTY

Constructor Summary

Constructors

Constructor	Description
CloudObjectInputSource(String scheme, List<URI> uris, List<URI> prefixes, List<CloudObjectLocation> objects, String objectGlob, SystemFields systemFields)

Method Summary

Modifier and Type	Method	Description
protected abstract InputEntity	createEntity(CloudObjectLocation location)	Create the correct InputEntity for this input source given a split on a CloudObjectLocation.
Stream<InputSplit<List<CloudObjectLocation>>>	createSplits(InputFormat inputFormat, SplitHintSpec splitHintSpec)	Creates a Stream of InputSplits.
boolean	equals(Object o)
int	estimateNumSplits(InputFormat inputFormat, SplitHintSpec splitHintSpec)	Returns an estimated total number of splits to be created via SplittableInputSource.createSplits(org.apache.druid.data.input.InputFormat, org.apache.druid.data.input.SplitHintSpec).
protected InputSourceReader	formattableReader(InputRowSchema inputRowSchema, InputFormat inputFormat, File temporaryDirectory)
Set<SystemField>	getConfiguredSystemFields()	System fields that this input source is configured to return.
String	getObjectGlob()
List<CloudObjectLocation>	getObjects()
List<URI>	getPrefixes()
protected abstract CloudObjectSplitWidget	getSplitWidget()	Returns CloudObjectSplitWidget, which is used to implement createSplits(InputFormat, SplitHintSpec).
List<URI>	getUris()
int	hashCode()
boolean	needsFormat()	Returns true if this inputSource supports different InputFormats.

Methods inherited from class org.apache.druid.data.input.AbstractInputSource

fixedFormatReader, reader

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.druid.data.input.InputSource

getTypes, reader

Methods inherited from interface org.apache.druid.data.input.impl.SplittableInputSource

getSplitHintSpecOrDefault, isSplittable, withSplit

Methods inherited from interface org.apache.druid.data.input.impl.systemfield.SystemFieldInputSource

getSystemFieldValue

Field Detail

systemFields

protected final SystemFields systemFields

Constructor Detail

CloudObjectInputSource

public CloudObjectInputSource(String scheme,
                              @Nullable
                              List<URI> uris,
                              @Nullable
                              List<URI> prefixes,
                              @Nullable
                              List<CloudObjectLocation> objects,
                              @Nullable
                              String objectGlob,
                              @Nullable
                              SystemFields systemFields)

Method Detail

getUris

public List<URI> getUris()

getPrefixes

public List<URI> getPrefixes()

getObjects

@Nullable
public List<CloudObjectLocation> getObjects()

getObjectGlob

@Nullable
public String getObjectGlob()

getConfiguredSystemFields

public Set<SystemField> getConfiguredSystemFields()

Description copied from interface: SystemFieldInputSource

System fields that this input source is configured to return. This is not the same set that SystemFieldInputSource.getSystemFieldValue(InputEntity, SystemField) returns nonnull for. For example, if a LocalInputSource is configured to return SystemField.BUCKET then it will show up in this list, even though its value is always null. For another example in a different direction, if a LocalInputSource is *not* configured to return SystemField.URI, then it will *not* show up in this list, even though its value from SystemFieldInputSource.getSystemFieldValue(InputEntity, SystemField) would be nonnull.

Specified by:

getConfiguredSystemFields in interface SystemFieldInputSource

createEntity

protected abstract InputEntity createEntity(CloudObjectLocation location)

Create the correct InputEntity for this input source given a split on a CloudObjectLocation. This is called internally by formattableReader(org.apache.druid.data.input.InputRowSchema, org.apache.druid.data.input.InputFormat, java.io.File) and operates on the output of createSplits(org.apache.druid.data.input.InputFormat, org.apache.druid.data.input.SplitHintSpec).

getSplitWidget

protected abstract CloudObjectSplitWidget getSplitWidget()

Returns CloudObjectSplitWidget, which is used to implement createSplits(InputFormat, SplitHintSpec).

createSplits

public Stream<InputSplit<List<CloudObjectLocation>>> createSplits(InputFormat inputFormat,
                                                                  @Nullable
                                                                  SplitHintSpec splitHintSpec)

Description copied from interface: SplittableInputSource

Creates a Stream of InputSplits. The returned stream is supposed to be evaluated lazily to avoid consuming too much memory. Note that this interface also has SplittableInputSource.estimateNumSplits(org.apache.druid.data.input.InputFormat, org.apache.druid.data.input.SplitHintSpec) which is related to this method. The implementations should be careful to NOT cache the created splits in memory. Implementations can consider InputFormat.isSplittable() and SplitHintSpec to create splits in the same way with SplittableInputSource.estimateNumSplits(org.apache.druid.data.input.InputFormat, org.apache.druid.data.input.SplitHintSpec).

Specified by:

createSplits in interface SplittableInputSource<List<CloudObjectLocation>>

estimateNumSplits

public int estimateNumSplits(InputFormat inputFormat,
                             @Nullable
                             SplitHintSpec splitHintSpec)

Description copied from interface: SplittableInputSource

Returns an estimated total number of splits to be created via SplittableInputSource.createSplits(org.apache.druid.data.input.InputFormat, org.apache.druid.data.input.SplitHintSpec). The estimated number of splits doesn't have to be accurate and can be different from the actual number of InputSplits returned from SplittableInputSource.createSplits(org.apache.druid.data.input.InputFormat, org.apache.druid.data.input.SplitHintSpec). This will be used to estimate the progress of a phase in parallel indexing. See TaskMonitor for more details of the progress estimation. This method can be expensive if an implementation iterates all directories or whatever substructure to find all input entities. Implementations can consider InputFormat.isSplittable() and SplitHintSpec to find splits in the same way with SplittableInputSource.createSplits(org.apache.druid.data.input.InputFormat, org.apache.druid.data.input.SplitHintSpec).

Specified by:

estimateNumSplits in interface SplittableInputSource<List<CloudObjectLocation>>

needsFormat

public boolean needsFormat()

Description copied from interface: InputSource

Returns true if this inputSource supports different InputFormats. Some inputSources such as LocalInputSource can store files of any format. These storage types require an InputFormat to be passed so that InputSourceReader can parse data properly. However, some storage types have a fixed format. For example, druid inputSource always reads segments. These inputSources should return false for this method.

Specified by:

needsFormat in interface InputSource

formattableReader

protected InputSourceReader formattableReader(InputRowSchema inputRowSchema,
                                              InputFormat inputFormat,
                                              @Nullable
                                              File temporaryDirectory)

Overrides:

formattableReader in class AbstractInputSource

equals

public boolean equals(Object o)

Overrides:

equals in class Object

hashCode

public int hashCode()

Overrides:

hashCode in class Object