Package org.apache.druid.data.input.impl
Class HttpInputSource
- java.lang.Object
-
- org.apache.druid.data.input.AbstractInputSource
-
- org.apache.druid.data.input.impl.HttpInputSource
-
- All Implemented Interfaces:
SplittableInputSource<URI>
,InputSource
public class HttpInputSource extends AbstractInputSource implements SplittableInputSource<URI>
-
-
Field Summary
Fields Modifier and Type Field Description static String
TYPE_KEY
-
Fields inherited from interface org.apache.druid.data.input.InputSource
TYPE_PROPERTY
-
Fields inherited from interface org.apache.druid.data.input.impl.SplittableInputSource
DEFAULT_SPLIT_HINT_SPEC
-
-
Constructor Summary
Constructors Constructor Description HttpInputSource(List<URI> uris, String httpAuthenticationUsername, PasswordProvider httpAuthenticationPasswordProvider, HttpInputSourceConfig config)
-
Method Summary
-
Methods inherited from class org.apache.druid.data.input.AbstractInputSource
fixedFormatReader, reader
-
Methods inherited from class java.lang.Object
clone, finalize, getClass, notify, notifyAll, wait, wait, wait
-
Methods inherited from interface org.apache.druid.data.input.InputSource
reader
-
Methods inherited from interface org.apache.druid.data.input.impl.SplittableInputSource
getSplitHintSpecOrDefault, isSplittable
-
-
-
-
Field Detail
-
TYPE_KEY
public static final String TYPE_KEY
- See Also:
- Constant Field Values
-
-
Constructor Detail
-
HttpInputSource
public HttpInputSource(List<URI> uris, @Nullable String httpAuthenticationUsername, @Nullable PasswordProvider httpAuthenticationPasswordProvider, HttpInputSourceConfig config)
-
-
Method Detail
-
getTypes
@Nonnull public Set<String> getTypes()
Description copied from interface:InputSource
The types of input sources uses. A set is returned here, as some InputSource implementation allow for combining of multiple input sources.- Specified by:
getTypes
in interfaceInputSource
- Returns:
- The types of input sources uses
-
throwIfInvalidProtocols
public static void throwIfInvalidProtocols(HttpInputSourceConfig config, List<URI> uris)
-
getHttpAuthenticationPasswordProvider
@Nullable public PasswordProvider getHttpAuthenticationPasswordProvider()
-
createSplits
public Stream<InputSplit<URI>> createSplits(InputFormat inputFormat, @Nullable SplitHintSpec splitHintSpec)
Description copied from interface:SplittableInputSource
Creates aStream
ofInputSplit
s. The returned stream is supposed to be evaluated lazily to avoid consuming too much memory. Note that this interface also hasSplittableInputSource.estimateNumSplits(org.apache.druid.data.input.InputFormat, org.apache.druid.data.input.SplitHintSpec)
which is related to this method. The implementations should be careful to NOT cache the created splits in memory. Implementations can considerInputFormat.isSplittable()
andSplitHintSpec
to create splits in the same way withSplittableInputSource.estimateNumSplits(org.apache.druid.data.input.InputFormat, org.apache.druid.data.input.SplitHintSpec)
.- Specified by:
createSplits
in interfaceSplittableInputSource<URI>
-
estimateNumSplits
public int estimateNumSplits(InputFormat inputFormat, @Nullable SplitHintSpec splitHintSpec)
Description copied from interface:SplittableInputSource
Returns an estimated total number of splits to be created viaSplittableInputSource.createSplits(org.apache.druid.data.input.InputFormat, org.apache.druid.data.input.SplitHintSpec)
. The estimated number of splits doesn't have to be accurate and can be different from the actual number of InputSplits returned fromSplittableInputSource.createSplits(org.apache.druid.data.input.InputFormat, org.apache.druid.data.input.SplitHintSpec)
. This will be used to estimate the progress of a phase in parallel indexing. See TaskMonitor for more details of the progress estimation. This method can be expensive if an implementation iterates all directories or whatever substructure to find all input entities. Implementations can considerInputFormat.isSplittable()
andSplitHintSpec
to find splits in the same way withSplittableInputSource.createSplits(org.apache.druid.data.input.InputFormat, org.apache.druid.data.input.SplitHintSpec)
.- Specified by:
estimateNumSplits
in interfaceSplittableInputSource<URI>
-
withSplit
public SplittableInputSource<URI> withSplit(InputSplit<URI> split)
Description copied from interface:SplittableInputSource
Helper method for ParallelIndexSupervisorTask. Most of implementations can simply create a new instance with the given split.- Specified by:
withSplit
in interfaceSplittableInputSource<URI>
-
formattableReader
protected InputSourceReader formattableReader(InputRowSchema inputRowSchema, InputFormat inputFormat, @Nullable File temporaryDirectory)
- Overrides:
formattableReader
in classAbstractInputSource
-
needsFormat
public boolean needsFormat()
Description copied from interface:InputSource
Returns true if this inputSource supports differentInputFormat
s. Some inputSources such asLocalInputSource
can store files of any format. These storage types require anInputFormat
to be passed so thatInputSourceReader
can parse data properly. However, some storage types have a fixed format. For example, druid inputSource always reads segments. These inputSources should return false for this method.- Specified by:
needsFormat
in interfaceInputSource
-
-