io.cdap.cdap.api.dataset.lib

Interface FileSet

All Superinterfaces:

AutoCloseable, Closeable, Dataset, InputFormatProvider, OutputFormatProvider

public interface FileSet
extends Dataset, InputFormatProvider, OutputFormatProvider

This dataset represents a collection of files on the file system. The dataset has a base location, under which all of its files are located. When instantiated, runtime arguments are required to specify the individual file or files being used.

Field Summary

Fields

Modifier and Type	Field and Description
static String	TYPE Type name

Method Summary

Modifier and Type	Method and Description
org.apache.twill.filesystem.Location	getBaseLocation() Allows to interact directly with the location of this dataset in the underlying file system.
Map<String,String>	getInputFormatConfiguration(Iterable<? extends org.apache.twill.filesystem.Location> inputLocs) A variant of InputFormatProvider.getInputFormatConfiguration() that allows passing in the input locations (rather than using the input locations that were determined from runtime arguments).
List<org.apache.twill.filesystem.Location>	getInputLocations() Allows direct access to files of this dataset in the underlying file system.
org.apache.twill.filesystem.Location	getLocation(String relativePath) Allows direct access to files in this dataset, in the underlying file system.
org.apache.twill.filesystem.Location	getOutputLocation() Allows direct access to files in the output location, in the underlying file system.
Map<String,String>	getRuntimeArguments() Allow direct access to the runtime arguments of this file set.

Methods inherited from interface java.io.Closeable

close

Methods inherited from interface io.cdap.cdap.api.data.batch.InputFormatProvider

getInputFormatClassName, getInputFormatConfiguration

Methods inherited from interface io.cdap.cdap.api.data.batch.OutputFormatProvider

getOutputFormatClassName, getOutputFormatConfiguration

Field Detail

TYPE

static final String TYPE

Type name

See Also:

Constant Field Values

Method Detail

getBaseLocation

org.apache.twill.filesystem.Location getBaseLocation()

Allows to interact directly with the location of this dataset in the underlying file system.

Returns:

the location of the base directory

getInputLocations

List<org.apache.twill.filesystem.Location> getInputLocations()

Allows direct access to files of this dataset in the underlying file system.

Returns:

the list of input locations

getOutputLocation

org.apache.twill.filesystem.Location getOutputLocation()

Allows direct access to files in the output location, in the underlying file system.

Returns:

the output location

getLocation

org.apache.twill.filesystem.Location getLocation(String relativePath)

Allows direct access to files in this dataset, in the underlying file system.

Returns:

the full location given by the path, relative to the base path.

getRuntimeArguments

Map<String,String> getRuntimeArguments()

Allow direct access to the runtime arguments of this file set.

Returns:

the runtime arguments specified for this file set.

getInputFormatConfiguration

Map<String,String> getInputFormatConfiguration(Iterable<? extends org.apache.twill.filesystem.Location> inputLocs)

A variant of InputFormatProvider.getInputFormatConfiguration() that allows passing in the input locations (rather than using the input locations that were determined from runtime arguments).

Parameters:

inputLocs - the input locations to be used