com.google.cloud.dataflow.sdk.util

Class IOChannelUtils

java.lang.Object

com.google.cloud.dataflow.sdk.util.IOChannelUtils

public class IOChannelUtils
extends Object

Provides utilities for creating read and write channels.

Constructor Summary

Constructors

Constructor and Description
IOChannelUtils()

Method Summary

Modifier and Type	Method and Description
static String	constructName(String prefix, String shardTemplate, String suffix, int shardNum, int numShards) Constructs a fully qualified name from components.
static WritableByteChannel	create(String filename, String mimeType) Creates a write channel for the given filename.
static WritableByteChannel	create(String prefix, String shardTemplate, String suffix, int numShards, String mimeType) Creates a write channel for the given file components.
static IOChannelFactory	getFactory(String spec) Returns the IOChannelFactory associated with an input specification.
static long	getSizeBytes(String spec) Returns the size in bytes for the given specification.
static void	registerStandardIOFactories(PipelineOptions options) Registers standard factories globally.
static String	resolve(String path, String other) Resolve the given other against the path.
static void	setIOFactory(String scheme, IOChannelFactory factory) Associates a scheme with an IOChannelFactory.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

IOChannelUtils

public IOChannelUtils()

Method Detail

setIOFactory

public static void setIOFactory(String scheme,
                                IOChannelFactory factory)

Associates a scheme with an IOChannelFactory.

The given factory is used to construct read and write channels when a URI is provided with the given scheme.

For example, when reading from "gs://bucket/path", the scheme "gs" is used to lookup the appropriate factory.

registerStandardIOFactories

public static void registerStandardIOFactories(PipelineOptions options)

Registers standard factories globally. This requires PipelineOptions to provide, e.g., credentials for GCS.

create

public static WritableByteChannel create(String filename,
                                         String mimeType)
                                  throws IOException

Creates a write channel for the given filename.

Throws:

IOException

create

public static WritableByteChannel create(String prefix,
                                         String shardTemplate,
                                         String suffix,
                                         int numShards,
                                         String mimeType)
                                  throws IOException

Creates a write channel for the given file components.

If numShards is specified, then a ShardingWritableByteChannel is returned.

Shard numbers are 0 based, meaning they start with 0 and end at the number of shards - 1.

Throws:

IOException

getSizeBytes

public static long getSizeBytes(String spec)
                         throws IOException

Returns the size in bytes for the given specification.

The specification is not expanded; it is used verbatim.

FileNotFoundException will be thrown if the resource does not exist.

Throws:

IOException

constructName

public static String constructName(String prefix,
                                   String shardTemplate,
                                   String suffix,
                                   int shardNum,
                                   int numShards)

Constructs a fully qualified name from components.

The name is built from a prefix, shard template (with shard numbers applied), and a suffix. All components are required, but may be empty strings.

Within a shard template, repeating sequences of the letters "S" or "N" are replaced with the shard number, or number of shards respectively. The numbers are formatted with leading zeros to match the length of the repeated sequence of letters.

For example, if prefix = "output", shardTemplate = "-SSS-of-NNN", and suffix = ".txt", with shardNum = 1 and numShards = 100, the following is produced: "output-001-of-100.txt".

getFactory

public static IOChannelFactory getFactory(String spec)
                                   throws IOException

Returns the IOChannelFactory associated with an input specification.

Throws:

IOException

resolve

public static String resolve(String path,
                             String other)
                      throws IOException

Resolve the given other against the path.

If the other parameter is an absolute path then this method trivially returns other. If other is an empty path then this method trivially returns the given path. Otherwise this method considers the given path to be a directory and resolves the other path against this path. In the simplest case, the other path does not have a root component, in which case this method joins the other path to the given path and returns a resulting path that ends with the other path. Where the other path has a root component then resolution is highly implementation dependent and therefore unspecified.

Throws:

IOException