io.cdap.cdap.api.dataset.lib

Class TimePartitionedFileSetArguments

java.lang.Object

io.cdap.cdap.api.dataset.lib.PartitionedFileSetArguments

io.cdap.cdap.api.dataset.lib.TimePartitionedFileSetArguments

@Beta
public class TimePartitionedFileSetArguments
extends PartitionedFileSetArguments

Helpers for manipulating runtime arguments of time-partitioned file sets.

Field Summary

Fields

Modifier and Type	Field and Description
static String	INPUT_END_TIME
static String	INPUT_START_TIME
static String	OUTPUT_PARTITION_TIME
static String	OUTPUT_PATH_FORMAT
static String	OUTPUT_TIME_ZONE

Fields inherited from class io.cdap.cdap.api.dataset.lib.PartitionedFileSetArguments

DYNAMIC_PARTITIONER_ALLOW_CONCURRENCY, DYNAMIC_PARTITIONER_CLASS_NAME, DYNAMIC_PARTITIONER_WRITE_OPTION, INPUT_PARTITION_FILTER, INPUT_PARTITION_LIST, OUTPUT_PARTITION_KEY_PREFIX, OUTPUT_PARTITION_METADATA_PREFIX

Constructor Summary

Constructors

Constructor and Description
TimePartitionedFileSetArguments()

Method Summary

Modifier and Type	Method and Description
static Long	getInputEndTime(Map<String,String> arguments) Get the end (exclusive) of the time range to be read as input.
static Long	getInputStartTime(Map<String,String> arguments) Get the start (inclusive) of the time range to be read as input.
static Long	getOutputPartitionTime(Map<String,String> arguments)
static String	getOutputPathFormat(Map<String,String> arguments) This is the file extension for each partition when using TimePartitionedFileSet as an OutputFormatProvider.
static String	getOutputPathTimeZone(Map<String,String> arguments) This is the time zone used to format the date for the output partition.
static void	setInputEndTime(Map<String,String> arguments, long time) Set the end (exclusive) of the time range to be read as input.
static void	setInputStartTime(Map<String,String> arguments, long time) Set the start (inclusive) of the time range to be read as input.
static void	setOutputPartitionTime(Map<String,String> arguments, long time) Set the time of the output partition when using TimePartitionedFileSet as an OutputFormatProvider.
static void	setOutputPathFormat(Map<String,String> arguments, String pathFormat) This is the file extension for each partition when using TimePartitionedFileSet as an OutputFormatProvider.
static void	setOutputPathFormat(Map<String,String> arguments, String pathFormat, String timeZone) This is the file extension for each partition when using TimePartitionedFileSet as an OutputFormatProvider.

Methods inherited from class io.cdap.cdap.api.dataset.lib.PartitionedFileSetArguments

addInputPartition, addInputPartitions, addInputPartitions, getDynamicPartitioner, getDynamicPartitionerWriteOption, getInputPartitionFilter, getInputPartitionKeys, getOutputPartitionKey, getOutputPartitionMetadata, isDynamicPartitionerConcurrencyAllowed, setDynamicPartitioner, setDynamicPartitioner, setDynamicPartitioner, setDynamicPartitioner, setDynamicPartitionerConcurrency, setInputPartitionFilter, setOutputPartitionKey, setOutputPartitionMetadata

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

OUTPUT_PATH_FORMAT

public static final String OUTPUT_PATH_FORMAT

See Also:

Constant Field Values

OUTPUT_TIME_ZONE

public static final String OUTPUT_TIME_ZONE

See Also:

Constant Field Values

OUTPUT_PARTITION_TIME

public static final String OUTPUT_PARTITION_TIME

See Also:

Constant Field Values

INPUT_START_TIME

public static final String INPUT_START_TIME

See Also:

Constant Field Values

INPUT_END_TIME

public static final String INPUT_END_TIME

See Also:

Constant Field Values

Constructor Detail

TimePartitionedFileSetArguments

public TimePartitionedFileSetArguments()

Method Detail

setOutputPartitionTime

public static void setOutputPartitionTime(Map<String,String> arguments,
                                          long time)

Set the time of the output partition when using TimePartitionedFileSet as an OutputFormatProvider. This time is used as the partition key for the new file, and also to generate an output file path - if that path is not explicitly given as an argument itself.

Parameters:

time - The time in milli seconds.

getOutputPartitionTime

@Nullable
public static Long getOutputPartitionTime(Map<String,String> arguments)

Returns:

the time of the output partition to be written

setOutputPathFormat

public static void setOutputPathFormat(Map<String,String> arguments,
                                       String pathFormat)

This is the file extension for each partition when using TimePartitionedFileSet as an OutputFormatProvider. It's used to generate the end of the output file path for each partition.

Parameters:

pathFormat - The format for the path; for example: 'yyyy-MM-dd/HH-mm,America/Los_Angeles', which will create a file path ending in the format of 2015-01-01/20-42, with the time of the partition being the time in the timezone of Los Angeles (PST or PDT). The pathFormat will be the format provided to SimpleDateFormat. If left blank, then the partitions will be of the form 2015-01-01/20-42.142017372000, with the time being the time UTC. Note that each partition must have a unique file path or a runtime exception will be thrown.

setOutputPathFormat

public static void setOutputPathFormat(Map<String,String> arguments,
                                       String pathFormat,
                                       @Nullable
                                       String timeZone)

This is the file extension for each partition when using TimePartitionedFileSet as an OutputFormatProvider. It's used to generate the end of the output file path for each partition.

Parameters:

pathFormat - The format for the path; for example: 'yyyy-MM-dd/HH-mm,America/Los_Angeles', which will create a file path ending in the format of 2015-01-01/20-42, with the time of the partition being the time in the timezone of Los Angeles (PST or PDT). The pathFormat will be the format provided to SimpleDateFormat. If left blank, then the partitions will be of the form 2015-01-01/20-42.142017372000, with the time being the time UTC. Note that each partition must have a unique file path or a runtime exception will be thrown.

timeZone - The string ID of the time zone. It is parsed by TimeZone.getTimeZone(String), and if the string ID is not a valid time zone, UTC is used.

getOutputPathFormat

@Nullable
public static String getOutputPathFormat(Map<String,String> arguments)

This is the file extension for each partition when using TimePartitionedFileSet as an OutputFormatProvider. It's used to generate the end of the output file path for each partition. May be null.

getOutputPathTimeZone

@Nullable
public static String getOutputPathTimeZone(Map<String,String> arguments)

This is the time zone used to format the date for the output partition.

Returns:

The String ID of the time zone of the date. May be null.

setInputStartTime

public static void setInputStartTime(Map<String,String> arguments,
                                     long time)

Set the start (inclusive) of the time range to be read as input.

Parameters:

time - The time in milli seconds.

getInputStartTime

@Nullable
public static Long getInputStartTime(Map<String,String> arguments)

Get the start (inclusive) of the time range to be read as input.

setInputEndTime

public static void setInputEndTime(Map<String,String> arguments,
                                   long time)

Set the end (exclusive) of the time range to be read as input.

Parameters:

time - The time in milli seconds.

getInputEndTime

public static Long getInputEndTime(Map<String,String> arguments)

Get the end (exclusive) of the time range to be read as input.