io.cdap.cdap.api.dataset.lib

Interface TimePartitionedFileSet

All Superinterfaces:

AutoCloseable, Closeable, Dataset, InputFormatProvider, OutputFormatProvider, PartitionedFileSet

@Beta
public interface TimePartitionedFileSet
extends PartitionedFileSet

Represents a dataset that is split into partitions that can be uniquely addressed by time. Each partition is a path in a file set, with a timestamp attached as meta data. The timestamp is mapped to a partition key of a PartitionedFileSet with five integer partitioning fields: the year, month, day, hour and minute. Partitions can be retrieved using time range or using a PartitionFilter.

The granularity of time is in minutes, that is, any seconds or milliseconds after the full minute is ignored for the partition keys. That means, there can not be be two partitions in the same minute. Also, when retrieving partitions via time or time range using getPartitionByTime(long), getPartitionsByTime(long, long), or when writing a partition using getPartitionOutput(long), the seconds and milliseconds on the time or time range are ignored.

This dataset can be made available for querying with SQL (explore). This is enabled through dataset properties when the dataset is created. See FileSetProperties for details. If it is enabled for explore, a Hive external table will be created when the dataset is created. The Hive table is partitioned by year, month, day, hour and minute.

Field Summary

Fields

Modifier and Type	Field and Description
static String	TYPE Type name

Method Summary

Modifier and Type	Method and Description
void	addMetadata(long time, Map<String,String> metadata) Adds a set of new metadata entries for a particular partition Note that existing entries can not be updated.
void	addMetadata(long time, String metadataKey, String metadataValue) Adds a new metadata entry for a particular partition.
void	addPartition(long time, String path) Add a partition for a given time, stored at a given path (relative to the file set's base path).
void	addPartition(long time, String path, Map<String,String> metadata) Add a partition for a given time, stored at a given path (relative to the file set's base path), with given metadata.
void	dropPartition(long time) Remove a partition for a given time.
TimePartitionDetail	getPartitionByTime(long time) Return the partition associated with the given time, rounded to the minute; or null if no such partition exists.
TimePartitionOutput	getPartitionOutput(long time) Return a partition output for a specific time, rounded to the minute, in preparation for creating a new partition.
Set<TimePartitionDetail>	getPartitionsByTime(long startTime, long endTime) Return all partitions within the time range given by startTime (inclusive) and endTime (exclusive), both rounded to the full minute.
void	removeMetadata(long time, Set<String> metadataKeys) Removes a set of metadata entries for a particular time.
void	removeMetadata(long time, String metadataKey) Removes a metadata entry for a particular time.
void	setMetadata(long time, Map<String,String> metadata) Sets metadata entries for a particular partition.

Methods inherited from interface io.cdap.cdap.api.dataset.lib.PartitionedFileSet

addMetadata, addMetadata, addPartition, addPartition, concatenatePartition, consumePartitions, consumePartitions, dropPartition, getEmbeddedFileSet, getPartition, getPartitioning, getPartitionOutput, getPartitions, getRuntimeArguments, removeMetadata, removeMetadata, setMetadata

Methods inherited from interface java.io.Closeable

close

Methods inherited from interface io.cdap.cdap.api.data.batch.InputFormatProvider

getInputFormatClassName, getInputFormatConfiguration

Methods inherited from interface io.cdap.cdap.api.data.batch.OutputFormatProvider

getOutputFormatClassName, getOutputFormatConfiguration

Field Detail

TYPE

static final String TYPE

Type name

See Also:

Constant Field Values

Method Detail

addPartition

void addPartition(long time,
                  String path)

Add a partition for a given time, stored at a given path (relative to the file set's base path).

Parameters:

time - the partition time in milliseconds since the Epoch

Throws:

PartitionAlreadyExistsException - if the partition for the given time already exists

addPartition

void addPartition(long time,
                  String path,
                  Map<String,String> metadata)

Add a partition for a given time, stored at a given path (relative to the file set's base path), with given metadata.

Parameters:

time - the partition time in milliseconds since the Epoch

Throws:

PartitionAlreadyExistsException - if the partition for the given time already exists

addMetadata

void addMetadata(long time,
                 String metadataKey,
                 String metadataValue)

Adds a new metadata entry for a particular partition. Note that existing entries can not be updated.

Parameters:

time - the partition time in milliseconds since the Epoch

Throws:

DataSetException - in case an attempt is made to update existing entries.

addMetadata

void addMetadata(long time,
                 Map<String,String> metadata)

Adds a set of new metadata entries for a particular partition Note that existing entries can not be updated.

Parameters:

time - the partition time in milliseconds since the Epoch

Throws:

DataSetException - in case an attempt is made to update existing entries.

setMetadata

void setMetadata(long time,
                 Map<String,String> metadata)

Sets metadata entries for a particular partition. If the metadata entry key does not already exist, it will be created; otherwise, it will be overwritten. Other existing keys remain unchanged.

Throws:

PartitionNotFoundException - when a partition for the given key is not found

IllegalArgumentException - if the partition key does not match the partitioning of the dataset

removeMetadata

void removeMetadata(long time,
                    String metadataKey)

Removes a metadata entry for a particular time. If the metadata key does not exist, no error is thrown.

Throws:

PartitionNotFoundException - when a partition for the given time is not found

removeMetadata

void removeMetadata(long time,
                    Set<String> metadataKeys)

Removes a set of metadata entries for a particular time. If any metadata key does not exist, no error is thrown.

Throws:

PartitionNotFoundException - when a partition for the given time is not found

dropPartition

void dropPartition(long time)

Remove a partition for a given time.

Parameters:

time - the partition time in milliseconds since the Epoch

getPartitionByTime

@Nullable
TimePartitionDetail getPartitionByTime(long time)

Return the partition associated with the given time, rounded to the minute; or null if no such partition exists.

Parameters:

time - the partition time in milliseconds since the Epoch

getPartitionsByTime

Set<TimePartitionDetail> getPartitionsByTime(long startTime,
                                             long endTime)

Return all partitions within the time range given by startTime (inclusive) and endTime (exclusive), both rounded to the full minute.

Parameters:

startTime - the inclusive lower bound for the partition time in milliseconds since the Epoch

endTime - the exclusive upper bound for the partition time in milliseconds since the Epoch

getPartitionOutput

TimePartitionOutput getPartitionOutput(long time)

Return a partition output for a specific time, rounded to the minute, in preparation for creating a new partition. Obtain the location to write from the PartitionOutput, then call the PartitionOutput.addPartition() to add the partition to this dataset.

Parameters:

time - the partition time in milliseconds since the Epoch

Throws:

PartitionAlreadyExistsException - if the partition for the given time already exists