com.google.cloud.dataflow.sdk.io

Class CompressedSource<T>

java.lang.Object

com.google.cloud.dataflow.sdk.io.Source<T>

com.google.cloud.dataflow.sdk.io.BoundedSource<T>

com.google.cloud.dataflow.sdk.io.ByteOffsetBasedSource<T>

com.google.cloud.dataflow.sdk.io.FileBasedSource<T>

com.google.cloud.dataflow.sdk.io.CompressedSource<T>

Type Parameters:

T - The type to read from the compressed file.

All Implemented Interfaces:

Serializable

@Experimental(value=SOURCE_SINK)
public class CompressedSource<T>
extends FileBasedSource<T>

A Source that reads from compressed files. A CompressedSources wraps a delegate FileBasedSource that is able to read the decompressed file format.

For example, use the following to read from a gzip-compressed XML file: XmlSource mySource = XmlSource.from(...); PCollection<T> collection = p.apply(CompressedSource.readFromSource(mySource, CompressedSource.CompressionMode.GZIP); Or, alternatively: XmlSource mySource = XmlSource.from(...); PCollection<T> collection = p.apply(Read.from(CompressedSource.from(mySource, CompressedSource.CompressionMode.GZIP)));

Default compression modes are CompressedSource.CompressionMode.GZIP and CompressedSource.CompressionMode.BZIP2. User-defined compression types are supported by implementing CompressedSource.DecompressingChannelFactory.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	CompressedSource.CompressedReader<T> Reader for a CompressedSource.
static class	CompressedSource.CompressionMode Default compression types supported by the CompressedSource.
static interface	CompressedSource.DecompressingChannelFactory Factory interface for creating channels that decompress the content of an underlying channel.

Nested classes/interfaces inherited from class com.google.cloud.dataflow.sdk.io.FileBasedSource

FileBasedSource.FileBasedReader<T>, FileBasedSource.Mode

Nested classes/interfaces inherited from class com.google.cloud.dataflow.sdk.io.ByteOffsetBasedSource

ByteOffsetBasedSource.ByteOffsetBasedReader<T>

Nested classes/interfaces inherited from class com.google.cloud.dataflow.sdk.io.BoundedSource

BoundedSource.AbstractBoundedReader<T>, BoundedSource.BoundedReader<T>

Nested classes/interfaces inherited from class com.google.cloud.dataflow.sdk.io.Source

Source.AbstractReader<T>, Source.Reader<T>

Method Summary

Modifier and Type	Method and Description
CompressedSource<T>	createForSubrangeOfFile(String fileName, long start, long end) Creates a CompressedSource for a subrange of a file.
CompressedSource.CompressedReader<T>	createSingleFileReader(PipelineOptions options, com.google.cloud.dataflow.sdk.util.ExecutionContext executionContext) Creates a CompressedReader to read a single file.
static <T> CompressedSource<T>	from(FileBasedSource<T> sourceDelegate) Creates a CompressedSource from an underlying FileBasedSource that must be further configured with withDecompression(com.google.cloud.dataflow.sdk.io.CompressedSource.DecompressingChannelFactory).
CompressedSource.DecompressingChannelFactory	getChannelFactory()
Coder<T>	getDefaultOutputCoder() Returns the delegate source's default output coder.
protected boolean	isSplittable() Determines whether a single file represented by this source is splittable.
boolean	producesSortedKeys(PipelineOptions options) Returns whether the delegate source produces sorted keys.
static <T> Read.Bound<T>	readFromSource(FileBasedSource<T> sourceDelegate, CompressedSource.DecompressingChannelFactory channelFactory) Creates a Read transform that reads from a CompressedSource that reads from an underlying FileBasedSource after decompressing it with a CompressedSource.DecompressingChannelFactory.
void	validate() Validates that the delegate source is a valid source and that the channel factory is not null.
CompressedSource<T>	withDecompression(CompressedSource.DecompressingChannelFactory channelFactory) Return a CompressedSource that is like this one but will decompress its underlying file with the given CompressedSource.DecompressingChannelFactory.

Methods inherited from class com.google.cloud.dataflow.sdk.io.FileBasedSource

createReader, createSourceForSubrange, getEstimatedSizeBytes, getFileOrPatternSpec, getMaxEndOffset, getMode, splitIntoBundles, toString

Methods inherited from class com.google.cloud.dataflow.sdk.io.ByteOffsetBasedSource

getEndOffset, getMinBundleSize, getStartOffset

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Method Detail

readFromSource

public static <T> Read.Bound<T> readFromSource(FileBasedSource<T> sourceDelegate,
                                               CompressedSource.DecompressingChannelFactory channelFactory)

Creates a Read transform that reads from a CompressedSource that reads from an underlying FileBasedSource after decompressing it with a CompressedSource.DecompressingChannelFactory.

from

public static <T> CompressedSource<T> from(FileBasedSource<T> sourceDelegate)

Creates a CompressedSource from an underlying FileBasedSource that must be further configured with withDecompression(com.google.cloud.dataflow.sdk.io.CompressedSource.DecompressingChannelFactory).

withDecompression

public CompressedSource<T> withDecompression(CompressedSource.DecompressingChannelFactory channelFactory)

Return a CompressedSource that is like this one but will decompress its underlying file with the given CompressedSource.DecompressingChannelFactory.

validate

public void validate()

Validates that the delegate source is a valid source and that the channel factory is not null.

Overrides:

validate in class FileBasedSource<T>

createForSubrangeOfFile

public CompressedSource<T> createForSubrangeOfFile(String fileName,
                                                   long start,
                                                   long end)

Creates a CompressedSource for a subrange of a file. Called by superclass to create a source for a single file.

Specified by:

createForSubrangeOfFile in class FileBasedSource<T>

Parameters:

fileName - file backing the new FileBasedSource.

start - starting byte offset of the new FileBasedSource.

end - ending byte offset of the new FileBasedSource. May be Long.MAX_VALUE, in which case it will be inferred using FileBasedSource.getMaxEndOffset(com.google.cloud.dataflow.sdk.options.PipelineOptions).

isSplittable

protected final boolean isSplittable()
                              throws Exception

Determines whether a single file represented by this source is splittable. Returns false: compressed sources are not splittable.

Overrides:

isSplittable in class FileBasedSource<T>

Throws:

Exception

createSingleFileReader

public final CompressedSource.CompressedReader<T> createSingleFileReader(PipelineOptions options,
                                                                         com.google.cloud.dataflow.sdk.util.ExecutionContext executionContext)

Creates a CompressedReader to read a single file.

Uses the delegate source to create a single file reader for the delegate source.

Specified by:

createSingleFileReader in class FileBasedSource<T>

producesSortedKeys

public final boolean producesSortedKeys(PipelineOptions options)
                                 throws Exception

Returns whether the delegate source produces sorted keys.

Specified by:

producesSortedKeys in class BoundedSource<T>

Throws:

Exception

getDefaultOutputCoder

public final Coder<T> getDefaultOutputCoder()

Returns the delegate source's default output coder.

Specified by:

getDefaultOutputCoder in class Source<T>

getChannelFactory

public final CompressedSource.DecompressingChannelFactory getChannelFactory()