io.cdap.cdap.api.dataset.lib

Interface FileSet

All Superinterfaces:

AutoCloseable, Closeable, Dataset, InputFormatProvider, OutputFormatProvider

public interface FileSet
extends Dataset, InputFormatProvider, OutputFormatProvider

This dataset represents a collection of files on the file system. The dataset has a base location, under which all of its files are located. When instantiated, runtime arguments are required to specify the individual file or files being used. This dataset can be made available for querying with SQL (explore). This is enabled through dataset properties when the dataset is created. See FileSetProperties for details. If it is enabled for explore, a Hive external table will be created when the dataset is created. The Hive table is not partitioned, and therefore querying will only work if every path in the fileset is a file (not a subdirectory).

Field Summary

Fields

Modifier and Type	Field and Description
static String	TYPE Type name

Method Summary

Modifier and Type	Method and Description
Location	getBaseLocation() Allows to interact directly with the location of this dataset in the underlying file system.
Map<String,String>	getInputFormatConfiguration(Iterable<? extends Location> inputLocs) A variant of InputFormatProvider.getInputFormatConfiguration() that allows passing in the input locations (rather than using the input locations that were determined from runtime arguments).
List<Location>	getInputLocations() Allows direct access to files of this dataset in the underlying file system.
Location	getLocation(String relativePath) Allows direct access to files in this dataset, in the underlying file system.
Location	getOutputLocation() Allows direct access to files in the output location, in the underlying file system.
Map<String,String>	getRuntimeArguments() Allow direct access to the runtime arguments of this file set.

Methods inherited from interface java.io.Closeable

close

Methods inherited from interface io.cdap.cdap.api.data.batch.InputFormatProvider

getInputFormatClassName, getInputFormatConfiguration

Methods inherited from interface io.cdap.cdap.api.data.batch.OutputFormatProvider

getOutputFormatClassName, getOutputFormatConfiguration

Field Detail

TYPE

static final String TYPE

Type name

See Also:

Constant Field Values

Method Detail

getBaseLocation

Location getBaseLocation()

Allows to interact directly with the location of this dataset in the underlying file system.

Returns:

the location of the base directory

getInputLocations

List<Location> getInputLocations()

Allows direct access to files of this dataset in the underlying file system.

Returns:

the list of input locations

getOutputLocation

Location getOutputLocation()

Allows direct access to files in the output location, in the underlying file system.

Returns:

the output location

getLocation

Location getLocation(String relativePath)

Allows direct access to files in this dataset, in the underlying file system.

Returns:

the full location given by the path, relative to the base path.

getRuntimeArguments

Map<String,String> getRuntimeArguments()

Allow direct access to the runtime arguments of this file set.

Returns:

the runtime arguments specified for this file set.

getInputFormatConfiguration

Map<String,String> getInputFormatConfiguration(Iterable<? extends Location> inputLocs)

A variant of InputFormatProvider.getInputFormatConfiguration() that allows passing in the input locations (rather than using the input locations that were determined from runtime arguments).

Parameters:

inputLocs - the input locations to be used