org.apache.beam.sdk.util.construction

Class UnboundedReadFromBoundedSource.BoundedToUnboundedSourceAdapter<T>

java.lang.Object

org.apache.beam.sdk.io.Source<OutputT>

org.apache.beam.sdk.io.UnboundedSource<T,UnboundedReadFromBoundedSource.BoundedToUnboundedSourceAdapter.Checkpoint<T>>

org.apache.beam.sdk.util.construction.UnboundedReadFromBoundedSource.BoundedToUnboundedSourceAdapter<T>

All Implemented Interfaces:

java.io.Serializable, HasDisplayData

Enclosing class:

UnboundedReadFromBoundedSource<T>

public static class UnboundedReadFromBoundedSource.BoundedToUnboundedSourceAdapter<T>
extends UnboundedSource<T,UnboundedReadFromBoundedSource.BoundedToUnboundedSourceAdapter.Checkpoint<T>>

A BoundedSource to UnboundedSource adapter.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	UnboundedReadFromBoundedSource.BoundedToUnboundedSourceAdapter.Checkpoint<T> A marker representing the progress and state of an UnboundedReadFromBoundedSource.BoundedToUnboundedSourceAdapter.

Nested classes/interfaces inherited from class org.apache.beam.sdk.io.UnboundedSource

UnboundedSource.CheckpointMark, UnboundedSource.UnboundedReader<OutputT>

Constructor Summary

Constructors

Constructor and Description
BoundedToUnboundedSourceAdapter(BoundedSource<T> boundedSource)

Method Summary

Modifier and Type	Method and Description
org.apache.beam.sdk.util.construction.UnboundedReadFromBoundedSource.BoundedToUnboundedSourceAdapter.Reader	createReader(PipelineOptions options, UnboundedReadFromBoundedSource.BoundedToUnboundedSourceAdapter.Checkpoint<T> checkpoint) Create a new UnboundedSource.UnboundedReader to read from this source, resuming from the given checkpoint if present.
Coder<UnboundedReadFromBoundedSource.BoundedToUnboundedSourceAdapter.Checkpoint<T>>	getCheckpointMarkCoder() Returns a Coder for encoding and decoding the checkpoints for this source.
Coder<T>	getDefaultOutputCoder()
java.util.List<UnboundedReadFromBoundedSource.BoundedToUnboundedSourceAdapter<T>>	split(int desiredNumSplits, PipelineOptions options) Returns a list of UnboundedSource objects representing the instances of this source that should be used when executing the workflow.
void	validate() Checks that this source is valid, before it can be used in a pipeline.

Methods inherited from class org.apache.beam.sdk.io.UnboundedSource

requiresDeduping

Methods inherited from class org.apache.beam.sdk.io.Source

getOutputCoder, populateDisplayData

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

BoundedToUnboundedSourceAdapter

public BoundedToUnboundedSourceAdapter(BoundedSource<T> boundedSource)

Method Detail

validate

public void validate()

Description copied from class: Source

Checks that this source is valid, before it can be used in a pipeline.

It is recommended to use Preconditions for implementing this method.

Overrides:

validate in class Source<T>

split

public java.util.List<UnboundedReadFromBoundedSource.BoundedToUnboundedSourceAdapter<T>> split(int desiredNumSplits,
                                                                                               PipelineOptions options)
                                                                                        throws java.lang.Exception

Description copied from class: UnboundedSource

Returns a list of UnboundedSource objects representing the instances of this source that should be used when executing the workflow. Each split should return a separate partition of the input data.

For example, for a source reading from a growing directory of files, each split could correspond to a prefix of file names.

Some sources are not splittable, such as reading from a single TCP stream. In that case, only a single split should be returned.

Some data sources automatically partition their data among readers. For these types of inputs, n identical replicas of the top-level source can be returned.

The size of the returned list should be as close to desiredNumSplits as possible, but does not have to match exactly. A low number of splits will limit the amount of parallelism in the source.

Specified by:

split in class UnboundedSource<T,UnboundedReadFromBoundedSource.BoundedToUnboundedSourceAdapter.Checkpoint<T>>

Throws:

java.lang.Exception

createReader

public org.apache.beam.sdk.util.construction.UnboundedReadFromBoundedSource.BoundedToUnboundedSourceAdapter.Reader createReader(PipelineOptions options,
                                                                                                                                UnboundedReadFromBoundedSource.BoundedToUnboundedSourceAdapter.Checkpoint<T> checkpoint)
                                                                                                                         throws java.io.IOException

Description copied from class: UnboundedSource

Create a new UnboundedSource.UnboundedReader to read from this source, resuming from the given checkpoint if present.

Specified by:

createReader in class UnboundedSource<T,UnboundedReadFromBoundedSource.BoundedToUnboundedSourceAdapter.Checkpoint<T>>

Throws:

java.io.IOException

getDefaultOutputCoder

public Coder<T> getDefaultOutputCoder()

Overrides:

getDefaultOutputCoder in class Source<T>

getCheckpointMarkCoder

public Coder<UnboundedReadFromBoundedSource.BoundedToUnboundedSourceAdapter.Checkpoint<T>> getCheckpointMarkCoder()

Description copied from class: UnboundedSource

Returns a Coder for encoding and decoding the checkpoints for this source.

Specified by:

getCheckpointMarkCoder in class UnboundedSource<T,UnboundedReadFromBoundedSource.BoundedToUnboundedSourceAdapter.Checkpoint<T>>