com.google.cloud.dataflow.sdk.io

Class FileBasedSource<T>

java.lang.Object

com.google.cloud.dataflow.sdk.io.Source<T>

com.google.cloud.dataflow.sdk.io.BoundedSource<T>

com.google.cloud.dataflow.sdk.io.ByteOffsetBasedSource<T>

com.google.cloud.dataflow.sdk.io.FileBasedSource<T>

Type Parameters:

T - Type of records represented by the source.

All Implemented Interfaces:

Serializable

Direct Known Subclasses:

BlockBasedSource, CompressedSource, XmlSource

public abstract class FileBasedSource<T>
extends ByteOffsetBasedSource<T>

A common base class for all file-based Sources. Extend this class to implement your own file-based custom source.

A file-based Source is a Source backed by a file pattern defined as a Java glob, a single file, or a offset range for a single file. See ByteOffsetBasedSource and RangeTracker for semantics of offset ranges.

This source stores a String that is an IOChannelFactory specification for a file or file pattern. There should be an IOChannelFactory defined for the file specification provided. Please refer to IOChannelUtils and IOChannelFactory for more information on this.

In addition to the methods left abstract from BoundedSource, subclasses must implement methods to create a sub-source and a reader for a range of a single file - createForSubrangeOfFile(java.lang.String, long, long) and createSingleFileReader(com.google.cloud.dataflow.sdk.options.PipelineOptions, com.google.cloud.dataflow.sdk.util.ExecutionContext). Please refer to XmlSource for an example implementation of FileBasedSource.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	FileBasedSource.FileBasedReader<T> A reader that implements code common to readers of FileBasedSources.
static class	FileBasedSource.Mode A given FileBasedSource represents a file resource of one of these types.

Nested classes/interfaces inherited from class com.google.cloud.dataflow.sdk.io.ByteOffsetBasedSource

ByteOffsetBasedSource.ByteOffsetBasedReader<T>

Nested classes/interfaces inherited from class com.google.cloud.dataflow.sdk.io.BoundedSource

BoundedSource.AbstractBoundedReader<T>, BoundedSource.BoundedReader<T>

Nested classes/interfaces inherited from class com.google.cloud.dataflow.sdk.io.Source

Source.AbstractReader<T>, Source.Reader<T>

Constructor Summary

Constructors

Constructor and Description
FileBasedSource(String fileOrPatternSpec, long minBundleSize) Create a FileBaseSource based on a file or a file pattern specification.
FileBasedSource(String fileName, long minBundleSize, long startOffset, long endOffset) Create a FileBasedSource based on a single file.

Method Summary

Modifier and Type	Method and Description
abstract FileBasedSource<T>	createForSubrangeOfFile(String fileName, long start, long end) Creates and returns a new FileBasedSource of the same type as the current FileBasedSource backed by a given file and an offset range.
BoundedSource.BoundedReader<T>	createReader(PipelineOptions options, com.google.cloud.dataflow.sdk.util.ExecutionContext executionContext)
abstract FileBasedSource.FileBasedReader<T>	createSingleFileReader(PipelineOptions options, com.google.cloud.dataflow.sdk.util.ExecutionContext executionContext) Creates and returns an instance of a FileBasedReader implementation for the current source assuming the source represents a single file.
FileBasedSource<T>	createSourceForSubrange(long start, long end) Returns a ByteOffsetBasedSource for a subrange of the current source.
long	getEstimatedSizeBytes(PipelineOptions options) An estimate of the total size (in bytes) of the data that would be read from this source.
String	getFileOrPatternSpec()
long	getMaxEndOffset(PipelineOptions options) Returns the exact ending offset of the current source.
FileBasedSource.Mode	getMode()
protected boolean	isSplittable() Determines whether a file represented by this source is can be split into bundles.
List<? extends FileBasedSource<T>>	splitIntoBundles(long desiredBundleSizeBytes, PipelineOptions options) Splits the source into bundles of approximately given size (in bytes).
String	toString()
void	validate() Checks that this source is valid, before it can be used in a pipeline.

Methods inherited from class com.google.cloud.dataflow.sdk.io.ByteOffsetBasedSource

getEndOffset, getMinBundleSize, getStartOffset

Methods inherited from class com.google.cloud.dataflow.sdk.io.BoundedSource

producesSortedKeys

Methods inherited from class com.google.cloud.dataflow.sdk.io.Source

getDefaultOutputCoder

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

FileBasedSource

public FileBasedSource(String fileOrPatternSpec,
                       long minBundleSize)

Create a FileBaseSource based on a file or a file pattern specification. This constructor must be used when creating a new FileBasedSource for a file pattern.

See ByteOffsetBasedSource for a detailed description of minBundleSize.

Parameters:

fileOrPatternSpec - IOChannelFactory specification of file or file pattern represented by the FileBasedSource.

minBundleSize - minimum bundle size in bytes.

FileBasedSource

public FileBasedSource(String fileName,
                       long minBundleSize,
                       long startOffset,
                       long endOffset)

Create a FileBasedSource based on a single file. This constructor must be used when creating a new FileBasedSource for a subrange of a single file. Additionally, this constructor must be used to create new FileBasedSources when subclasses implement the method createForSubrangeOfFile(java.lang.String, long, long).

See ByteOffsetBasedSource for detailed descriptions of minBundleSize, startOffset, and endOffset.

Parameters:

fileName - IOChannelFactory specification of the file represented by the FileBasedSource.

minBundleSize - minimum bundle size in bytes.

startOffset - starting byte offset.

endOffset - ending byte offset. If the specified value >= #getMaxEndOffset() it implies #getMaxEndOffSet().

Method Detail

getFileOrPatternSpec

public final String getFileOrPatternSpec()

getMode

public final FileBasedSource.Mode getMode()

createSourceForSubrange

public final FileBasedSource<T> createSourceForSubrange(long start,
                                                        long end)

Description copied from class: ByteOffsetBasedSource

Returns a ByteOffsetBasedSource for a subrange of the current source. [start, end) will be within the range [startOffset, endOffset] of the current source.

Specified by:

createSourceForSubrange in class ByteOffsetBasedSource<T>

createForSubrangeOfFile

public abstract FileBasedSource<T> createForSubrangeOfFile(String fileName,
                                                           long start,
                                                           long end)

Creates and returns a new FileBasedSource of the same type as the current FileBasedSource backed by a given file and an offset range. When current source is being split, this method is used to generate new sub-sources. When creating the source subclasses must call the constructor FileBasedSource(String, long, long, long) of FileBasedSource with corresponding parameter values passed here.

Parameters:

fileName - file backing the new FileBasedSource.

start - starting byte offset of the new FileBasedSource.

end - ending byte offset of the new FileBasedSource. May be Long.MAX_VALUE, in which case it will be inferred using getMaxEndOffset(com.google.cloud.dataflow.sdk.options.PipelineOptions).

createSingleFileReader

public abstract FileBasedSource.FileBasedReader<T> createSingleFileReader(PipelineOptions options,
                                                                          com.google.cloud.dataflow.sdk.util.ExecutionContext executionContext)

Creates and returns an instance of a FileBasedReader implementation for the current source assuming the source represents a single file. File patterns will be handled by FileBasedSource implementation automatically.

getEstimatedSizeBytes

public final long getEstimatedSizeBytes(PipelineOptions options)
                                 throws Exception

Description copied from class: BoundedSource

An estimate of the total size (in bytes) of the data that would be read from this source. This estimate is in terms of external storage size, before any decompression or other processing done by the reader.

Specified by:

getEstimatedSizeBytes in class BoundedSource<T>

Throws:

Exception

splitIntoBundles

public final List<? extends FileBasedSource<T>> splitIntoBundles(long desiredBundleSizeBytes,
                                                                 PipelineOptions options)
                                                          throws Exception

Description copied from class: BoundedSource

Splits the source into bundles of approximately given size (in bytes).

Overrides:

splitIntoBundles in class ByteOffsetBasedSource<T>

Throws:

Exception

isSplittable

protected boolean isSplittable()
                        throws Exception

Determines whether a file represented by this source is can be split into bundles.

By default, a file is splittable if it is on a file system that supports efficient read seeking. Subclasses may override to provide different behavior.

Throws:

Exception

createReader

public final BoundedSource.BoundedReader<T> createReader(PipelineOptions options,
                                                         com.google.cloud.dataflow.sdk.util.ExecutionContext executionContext)
                                                  throws IOException

Specified by:

createReader in class BoundedSource<T>

Throws:

IOException

toString

public String toString()

Overrides:

toString in class ByteOffsetBasedSource<T>

validate

public void validate()

Description copied from class: Source

Checks that this source is valid, before it can be used in a pipeline.

It is recommended to use Preconditions for implementing this method.

Overrides:

validate in class ByteOffsetBasedSource<T>

getMaxEndOffset

public final long getMaxEndOffset(PipelineOptions options)
                           throws Exception

Description copied from class: ByteOffsetBasedSource

Returns the exact ending offset of the current source. This will be used if the source was constructed with an endOffset value Long.MAX_VALUE.

Specified by:

getMaxEndOffset in class ByteOffsetBasedSource<T>

Throws:

Exception