All Classes

Class	Description
AbstractFileSource<T,SplitT extends FileSourceSplit>	The base class for File Sources.
AbstractFileSource.AbstractFileSourceBuilder<T,SplitT extends FileSourceSplit,SELF extends AbstractFileSource.AbstractFileSourceBuilder<T,SplitT,SELF>>	The generic base builder.
AbstractStreamingWriter<IN,OUT>	Operator for file system sink.
ArrayResultIterator<E>	A simple BulkFormat.RecordIterator that returns the elements of an array, one after the other.
BatchCompactCoordinator	Coordinator for compaction in batch mode.
BatchCompactOperator<T>	CompactOperator for compaction in batch mode.
BatchFileWriter<T>	An operator for writing files in batch mode.
BatchPartitionCommitterSink	Committer operator for partition in batch mode.
BatchSink	Helper for creating batch file sink.
BinPacking	A bin packing implementation.
BlockSplittingRecursiveAllDirEnumerator	This FileEnumerator enumerates all files under the given paths recursively except the hidden directories, and creates a separate split for each file block.
BlockSplittingRecursiveEnumerator	This FileEnumerator enumerates all files under the given paths recursively, and creates a separate split for each file block.
BulkDecodingFormat<T>	A Format for a DynamicTableSource for reading rows by BulkFormat.
BulkFormat<T,SplitT extends FileSourceSplit>	The BulkFormat reads and decodes batches of records at a time.
BulkFormat.Reader<T>	The actual reader that reads the batches of records.
BulkFormat.RecordIterator<T>	An iterator over records with their position in the file.
BulkReaderFormatFactory	Base interface for configuring a BulkFormat for file system connector.
BulkWriterFormatFactory	Base interface for configuring a BulkWriter.Factory for file system connector.
CheckpointedPosition	The position of a reader, to be stored in a checkpoint.
ColumnarRowIterator	A BulkFormat.RecordIterator that returns RowDatas.
CompactBucketWriter<T>	The CompactWriter to delegate BucketWriter.
CompactBulkReader<T>	The CompactReader to delegate CompactBulkReader.
CompactContext	Context for CompactReader and CompactWriter.
CompactContext.CompactContextImpl	Implementation of CompactContext.
CompactCoordinator	Coordinator that coordinates file compaction for the FileSink.
CompactCoordinator	This is the single (non-parallel) monitoring task which coordinate input files to compaction units. - Receives in-flight input files inside checkpoint. - Receives all upstream end input messages after the checkpoint completes successfully, starts coordination.
CompactCoordinatorFactory	Factory for CompactCoordinator.
CompactCoordinatorStateHandler	Handler that processes the state of CompactCoordinator when compaction is disabled.
CompactCoordinatorStateHandlerFactory	Factory for CompactCoordinatorStateHandler.
CompactFileUtils	Utils for compacting files.
CompactFileWriter<T>	Writer for emitting CompactMessages.InputFile and CompactMessages.EndCheckpoint to downstream.
CompactMessages	Util class for all compaction messages.
CompactMessages.CompactionUnit	The unit of a single compaction.
CompactMessages.CompactOutput	The output of BatchCompactOperator.
CompactMessages.CoordinatorInput	The input of compact coordinator.
CompactMessages.CoordinatorOutput	The output of compact coordinator.
CompactMessages.EndCheckpoint	A flag to end checkpoint, coordinator can start coordinating one checkpoint.
CompactMessages.EndCompaction	A flag to end compaction.
CompactMessages.InputFile	A partitioned input file.
CompactOperator<T>	Receives compaction units to do compaction.
CompactorOperator	An operator that perform compaction for the FileSink.
CompactorOperatorFactory	Factory for CompactorOperator.
CompactorOperatorStateHandler	Handler that processes the state of CompactorOperator when compaction is disabled.
CompactorOperatorStateHandlerFactory	Factory for CompactorOperatorStateHandler.
CompactorRequest	Request of file compacting for FileSink.
CompactorRequestSerializer	Versioned serializer for CompactorRequest.
CompactorRequestTypeInfo	Type information of CompactorRequest.
CompactReader<T>	Reader for compaction.
CompactReader.Factory<T>	Factory to create CompactReader.
CompactService	The asynchronous file compaction service.
CompactWriter<T>	Writer for compaction.
CompactWriter.Factory<T>	Factory to create CompactWriter.
ConcatFileCompactor	A OutputStreamBasedFileCompactor implementation that simply concat the compacting files.
ContinuousEnumerationSettings	Settings describing how to do continuous file discovery and enumeration for the file source's continuous discovery and streaming mode.
ContinuousFileSplitEnumerator	A continuously monitoring enumerator.
ContinuousPartitionFetcher<P,T extends Comparable<T>>	Partition fetcher for helping continuously fetch partitioned table.
ContinuousPartitionFetcher.Context<P,T extends Comparable<T>>	Context for fetch partitions, partition information is stored in hive meta store.
DecoderBasedReader<T>	A RecordWiseFileCompactor.Reader implementation that reads the file as an FSDataInputStream and decodes the record with the DecoderBasedReader.Decoder.
DecoderBasedReader.Decoder<T>	A DecoderBasedReader.Decoder to decode the file content into the actual records.
DecoderBasedReader.Decoder.Factory<T>	Factory to create DecoderBasedReader.Decoder.
DecoderBasedReader.Factory<T>	Factory for DecoderBasedReader.
DefaultFileFilter	A file filter that filters out hidden files based on common naming patterns, i.e., files where the filename starts with '.' or with '_'.
DefaultFileWriterBucketFactory<IN>	A factory returning writer.
DefaultPartTimeExtractor	Default PartitionTimeExtractor.
DeserializationSchemaAdapter	Adapter to turn a DeserializationSchema into a BulkFormat.
DynamicFileEnumerator	FileEnumerator that supports dynamic filtering.
DynamicFileEnumerator.Provider	Factory for the DynamicFileEnumerator.
DynamicFileSplitEnumerator<SplitT extends FileSourceSplit>	A SplitEnumerator implementation that supports dynamic filtering.
DynamicPartitionWriter<T>	Dynamic partition writer to writing multiple partitions at the same time, it maybe consumes more memory.
EmptyMetaStoreFactory	Empty implementation TableMetaStoreFactory.
EnrichedRowData	An implementation of RowData which is backed by two RowData with a well-defined index mapping, One of the rows is fixed, while the other can be swapped for performant changes in hot code paths.
FileCommitter	Committer implementation for FileSink.
FileCompactor	The FileCompactor is responsible for compacting files into one file.
FileCompactStrategy	Strategy for compacting the files written in FileSink before committing.
FileCompactStrategy.Builder	Builder for FileCompactStrategy.
FileEnumerator	The FileEnumerator's task is to discover all files to be read and to split them into a set of FileSourceSplit.
FileEnumerator.Provider	Factory for the FileEnumerator, to allow the FileEnumerator to be eagerly initialized and to not be serializable.
FileInputFormatCompactReader<T>	The CompactReader to delegate FileInputFormat.
FileRecords<T>	A collection of records for one file split.
FileSink<IN>	A unified sink that emits its input elements to FileSystem files within buckets.
FileSink.BulkFormatBuilder<IN,T extends FileSink.BulkFormatBuilder<IN,T>>	A builder for configuring the sink for bulk-encoding formats, e.g.
FileSink.DefaultBulkFormatBuilder<IN>	Builder for the vanilla FileSink using a bulk format.
FileSink.DefaultRowFormatBuilder<IN>	Builder for the vanilla FileSink using a row format.
FileSink.RowFormatBuilder<IN,T extends FileSink.RowFormatBuilder<IN,T>>	A builder for configuring the sink for row-wise encoding formats.
FileSinkCommittable	Wrapper class for both type of committables in FileSink.
FileSinkCommittableSerializer	Versioned serializer for FileSinkCommittable.
FileSource<T>	A unified data source that reads files - both in batch and in streaming mode.
FileSource.FileSourceBuilder<T>	The builder for the FileSource, to configure the various behaviors.
FileSourceReader<T,SplitT extends FileSourceSplit>	A SourceReader that read records from FileSourceSplit.
FileSourceSplit	A SourceSplit that represents a file, or a region of a file.
FileSourceSplitSerializer	A serializer for the FileSourceSplit.
FileSourceSplitState<SplitT extends FileSourceSplit>	State of the reader, essentially a mutable version of the FileSourceSplit.
FileSplitAssigner	The FileSplitAssigner is responsible for deciding what split should be processed next by which node.
FileSplitAssigner.Provider	Factory for the FileSplitAssigner, to allow the FileSplitAssigner to be eagerly initialized and to not be serializable.
FileSystemCommitter	File system file committer implementation.
FileSystemConnectorOptions	Options for the filesystem connector.
FileSystemConnectorOptions.FileStatisticsType	Statistics types for file system, see FileSystemConnectorOptions.SOURCE_REPORT_STATISTICS.
FileSystemConnectorOptions.PartitionCommitTriggerType	Trigger types for partition commit, see FileSystemConnectorOptions.SINK_PARTITION_COMMIT_TRIGGER.
FileSystemFactory	A factory to create file systems.
FileSystemOutputFormat<T>	File system OutputFormat for batch job.
FileSystemOutputFormat.Builder<T>	Builder to build FileSystemOutputFormat.
FileSystemTableFactory	File system TableFactory.
FileSystemTableSink	File system DynamicTableSink.
FileSystemTableSink.ProjectionBulkFactory	Project row to non-partition fields.
FileSystemTableSink.TableBucketAssigner	Table bucket assigner, wrap PartitionComputer.
FileSystemTableSink.TableRollingPolicy	Table RollingPolicy, it extends CheckpointRollingPolicy for bulk writers.
FileSystemTableSource	File system table source.
FileWriter<IN>	A SinkWriter implementation for FileSink.
FileWriterBucketFactory<IN>	A factory able to create FileWriterBucket for the FileSink.
FileWriterBucketState	States for FileWriterBucket.
FileWriterBucketStateSerializer	A SimpleVersionedSerializer used to serialize the BucketState.
GroupedPartitionWriter<T>	PartitionWriter for grouped dynamic partition inserting.
IdenticalFileCompactor	A simple OutputStreamBasedFileCompactor implementation that directly copy the content of the only input file to the output.
InputFormatBasedReader<T>	A RecordWiseFileCompactor.Reader implementation that reads the file using the FileInputFormat.
InputFormatBasedReader.Factory<T>	Factory for InputFormatBasedReader.
IteratorResultIterator<E>	A simple BulkFormat.RecordIterator that returns the elements of an iterator, augmented with position information.
LimitableBulkFormat<T,SplitT extends FileSourceSplit>	A BulkFormat that can limit output record number.
LocalityAwareSplitAssigner	A FileSplitAssigner that assigns to each host preferably splits that are local, before assigning splits that are not local.
MetastoreCommitPolicy	Partition commit policy to update metastore.
MutableRecordAndPosition<E>	A mutable version of the RecordAndPosition.
NonSplittingRecursiveAllDirEnumerator	This FileEnumerator enumerates all files under the given paths recursively except the hidden directories.
NonSplittingRecursiveEnumerator	This FileEnumerator enumerates all files under the given paths recursively.
OutputFormatFactory<T>	A factory to create an OutputFormat.
OutputStreamBasedFileCompactor	Base class for FileCompactor implementations that write the compacting file by a output stream.
PartitionCommitInfo	The message sent by upstream.
PartitionCommitPolicy	Policy for commit a partition.
PartitionCommitPolicy.Context	Context of policy, including table information and partition information.
PartitionCommitPolicyFactory	A factory to create PartitionCommitPolicy chain.
PartitionCommitPredicate	Partition commit predicate.
PartitionCommitPredicate.PredicateContext	Context that PartitionCommitPredicate can use for getting context about a partition.
PartitionCommitter	Committer operator for partitions.
PartitionCommitTrigger	Partition commit trigger.
PartitionComputer<T>	Compute partition path from record and project non-partition columns for output writer.
PartitionFetcher<P>	Fetcher to fetch the suitable partitions of a filesystem table.
PartitionFetcher.Context<P>	Context for fetch partitions, partition information is stored in hive meta store.
PartitionFetcher.Context.ComparablePartitionValue<P,T extends Comparable<T>>	A comparable partition value that can compare order by using its comparator.
PartitionFieldExtractor<T extends FileSourceSplit>	Interface to extract partition field from split.
PartitionLoader	Loader to temporary files to final output path and meta store.
PartitionReader<P,OUT>	Reader that reads record from given partitions.
PartitionTempFileManager	Manage temporary files for writing files.
PartitionTimeCommitPredicate	Partition commit predicate by partition time and watermark, if 'watermark' > 'partition-time' + 'delay', the partition is committable.
PartitionTimeCommitTrigger	Partition commit trigger by partition time and watermark.
PartitionTimeExtractor	Time extractor to extract time from partition values.
PartitionWriter<T>	Partition writer to write records with partition.
PartitionWriter.Context<T>	Context for partition writer, provide some information and generation utils.
PartitionWriter.DefaultPartitionWriterListener	Default implementation for PartitionWriterListener.
PartitionWriter.PartitionWriterListener	Listener for partition writer.
PartitionWriterFactory<T>	Factory of PartitionWriter to avoid virtual function calls.
PendingSplitsCheckpoint<SplitT extends FileSourceSplit>	A checkpoint of the current state of the containing the currently pending splits that are not yet assigned.
PendingSplitsCheckpointSerializer<T extends FileSourceSplit>	A serializer for the PendingSplitsCheckpoint.
Pool<T>	A pool to cache and recycle heavyweight objects, to reduce object allocation.
Pool.Recycler<T>	A Recycler puts objects into the pool that the recycler is associated with.
ProcTimeCommitPredicate	Partition commit trigger by creation time and processing time service, if 'current processing time' > 'partition creation time' + 'delay', the partition is committable.
ProcTimeCommitTrigger	Partition commit trigger by creation time and processing time service.
RecordAndPosition<E>	A record, together with the reader position to be stored in the checkpoint.
RecordMapperWrapperRecordIterator<I,O>	Implementation of BulkFormat.RecordIterator that wraps another iterator and performs the mapping of the records.
RecordMapperWrapperRecordIterator.RecordMapper<I,O>	Record mapper definition.
RecordWiseFileCompactor<IN>	A FileCompactor implementation that reads input files with a RecordWiseFileCompactor.Reader and writes with a RecordWiseFileCompactor.Writer.
RecordWiseFileCompactor.Reader<T>	The reader that reads record from the compacting files.
RecordWiseFileCompactor.Reader.Factory<T>	Factory for RecordWiseFileCompactor.Reader.
RecordWiseFileCompactor.Writer<T>	The writer that writers record into the compacting files.
RecyclableIterator<E>	Utility base class for iterators that accept a recycler.
RegexFileFilter	A file filter that filters out hidden files, see DefaultFileFilter and the files whose path doesn't match the given regex pattern.
RowDataPartitionComputer	PartitionComputer for RowData.
RowPartitionComputer	PartitionComputer for Row.
SerializationSchemaAdapter	Adapter to turn a SerializationSchema into a Encoder.
SimpleSplitAssigner	The SimpleSplitAssigner hands out splits in a random order, without any consideration for order or locality.
SimpleStreamFormat<T>	A simple version of the StreamFormat, for formats that are not splittable.
SimpleStringDecoder	A sink DecoderBasedReader.Decoder that reads data encoded by the SimpleStringEncoder only for compaction.
SingleDirectoryWriter<T>	PartitionWriter for single directory writer.
SingletonResultIterator<E>	A simple BulkFormat.RecordIterator that returns a single value.
StandardDeCompressors	A collection of common compression formats and de-compressors.
StaticFileSplitEnumerator	A SplitEnumerator implementation for bounded / batch FileSource input.
StreamFormat<T>	A reader format that reads individual records from a stream.
StreamFormat.Reader<T>	The actual reader that reads the records.
StreamFormatAdapter<T>	Adapter to turn a StreamFormat into a BulkFormat.
StreamFormatAdapter.Reader<T>	The reader adapter, from StreamFormat.Reader to BulkFormat.Reader.
StreamingFileWriter<IN>	Writer for emitting PartitionCommitInfo to downstream.
StreamingSink	Helper for creating streaming file sink.
SuccessFileCommitPolicy	Partition commit policy to add success file to directory.
TableMetaStoreFactory	Meta store factory to create TableMetaStoreFactory.TableMetaStore.
TableMetaStoreFactory.TableMetaStore	Meta store to manage the location paths of this table and its partitions.
TaskTracker	Track the upstream tasks to determine whether all the upstream data of a checkpoint has been received.
TextLineInputFormat	A reader format that text lines from a file.
TextLineInputFormat.Reader	The actual reader for the TextLineInputFormat.
Utils	Miscellaneous utilities for the file source.