org.apache.hadoop.fs.s3a.commit

Class CommitConstants

java.lang.Object

org.apache.hadoop.fs.s3a.commit.CommitConstants

@InterfaceAudience.Public
 @InterfaceStability.Unstable
public final class CommitConstants
extends Object

Constants for working with committers.

Field Summary

Fields

Modifier and Type	Field and Description
static String	_SUCCESS Marker file to create on success: "_SUCCESS".
static String	BASE Marker of the start of a directory tree for calculating the final path names: "__base".
static String	COMMITTER_NAME_DIRECTORY Option for FS_S3A_COMMITTER_NAME: directory output committer: "directory".
static String	COMMITTER_NAME_FILE Option for FS_S3A_COMMITTER_NAME: classic/file output committer: "file".
static String	COMMITTER_NAME_MAGIC Option for FS_S3A_COMMITTER_NAME: magic output committer: "magic".
static String	COMMITTER_NAME_PARTITIONED Option for FS_S3A_COMMITTER_NAME: partition output committer: "partitioned".
static String	CONFLICT_MODE_APPEND Conflict mode: "append".
static String	CONFLICT_MODE_FAIL Conflict mode: "fail".
static String	CONFLICT_MODE_REPLACE Conflict mode: "replace".
static String	CREATE_SUCCESSFUL_JOB_OUTPUT_DIR_MARKER Flag to trigger creation of a marker file on job completion.
static int	DEFAULT_COMMITTER_THREADS Default value for FS_S3A_COMMITTER_THREADS: 8.
static String	DEFAULT_CONFLICT_MODE Default conflict mode: "fail".
static boolean	DEFAULT_CREATE_SUCCESSFUL_JOB_DIR_MARKER Default job marker option: true.
static boolean	DEFAULT_MAGIC_COMMITTER_ENABLED Is the committer enabled by default? No.
static boolean	DEFAULT_STAGING_COMMITTER_UNIQUE_FILENAMES Default value for FS_S3A_COMMITTER_STAGING_UNIQUE_FILENAMES: true.
static String	FS_S3A_COMMITTER_NAME Option to identify the S3A committer: "fs.s3a.committer.name".
static String	FS_S3A_COMMITTER_STAGING_ABORT_PENDING_UPLOADS Should the staging committers abort all pending uploads to the destination directory? Default: true.
static String	FS_S3A_COMMITTER_STAGING_CONFLICT_MODE Staging committer conflict resolution policy: "fs.s3a.committer.staging.conflict-mode".
static String	FS_S3A_COMMITTER_STAGING_TMP_PATH Path in the cluster filesystem for temporary data: "fs.s3a.committer.staging.tmp.path".
static String	FS_S3A_COMMITTER_STAGING_UNIQUE_FILENAMES Option for final files to have a uniqueness name through job attempt info, falling back to a new UUID if there is no job attempt information to use.
static String	FS_S3A_COMMITTER_THREADS Number of threads in committers for parallel operations on files (upload, commit, abort, delete...): "fs.s3a.committer.threads".
static String	MAGIC Path for "magic" writes: path and PENDING_SUFFIX files: "__magic".
static String	MAGIC_COMMITTER_ENABLED Flag to indicate whether support for the Magic committer is enabled in the filesystem.
static String	MAGIC_COMMITTER_PREFIX Flag to indicate whether support for the Magic committer is enabled in the filesystem.
static String	PENDING_SUFFIX Suffix applied to pending commit metadata: ".pending".
static String	PENDINGSET_SUFFIX Suffix applied to multiple pending commit metadata: ".pendingset".
static String	S3A_COMMITTER_FACTORY S3 Committer factory: "org.apache.hadoop.fs.s3a.commit.S3ACommitterFactory".
static String	S3A_COMMITTER_FACTORY_KEY Key to set for the S3A schema to use the specific committer.
static String	STORE_CAPABILITY_MAGIC_COMMITTER Flag to indicate that a store supports magic committers.
static String	STREAM_CAPABILITY_MAGIC_OUTPUT Flag to indicate whether a stream is a magic output stream; returned in StreamCapabilities Value: "s3a:magic.output.stream".
static String	TEMP_DATA Temp data which is not auto-committed: "__temp-data".
static String	TEMPORARY This is the "Pending" directory of the FileOutputCommitter; data written here is, in that algorithm, renamed into place.

Method Summary

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

MAGIC

public static final String MAGIC

Path for "magic" writes: path and PENDING_SUFFIX files: "__magic".

See Also:

Constant Field Values

BASE

public static final String BASE

Marker of the start of a directory tree for calculating the final path names: "__base".

See Also:

Constant Field Values

PENDING_SUFFIX

public static final String PENDING_SUFFIX

Suffix applied to pending commit metadata: ".pending".

See Also:

Constant Field Values

PENDINGSET_SUFFIX

public static final String PENDINGSET_SUFFIX

Suffix applied to multiple pending commit metadata: ".pendingset".

See Also:

Constant Field Values

MAGIC_COMMITTER_PREFIX

public static final String MAGIC_COMMITTER_PREFIX

Flag to indicate whether support for the Magic committer is enabled in the filesystem. Value: "fs.s3a.committer.magic".

See Also:

Constant Field Values

MAGIC_COMMITTER_ENABLED

public static final String MAGIC_COMMITTER_ENABLED

Flag to indicate whether support for the Magic committer is enabled in the filesystem. Value: "fs.s3a.committer.magic.enabled".

See Also:

Constant Field Values

STREAM_CAPABILITY_MAGIC_OUTPUT

public static final String STREAM_CAPABILITY_MAGIC_OUTPUT

Flag to indicate whether a stream is a magic output stream; returned in StreamCapabilities Value: "s3a:magic.output.stream".

See Also:

Constant Field Values

STORE_CAPABILITY_MAGIC_COMMITTER

public static final String STORE_CAPABILITY_MAGIC_COMMITTER

Flag to indicate that a store supports magic committers. returned in StreamCapabilities Value: "s3a:magic.committer".

See Also:

Constant Field Values

DEFAULT_MAGIC_COMMITTER_ENABLED

public static final boolean DEFAULT_MAGIC_COMMITTER_ENABLED

Is the committer enabled by default? No.

See Also:

Constant Field Values

TEMPORARY

public static final String TEMPORARY

This is the "Pending" directory of the FileOutputCommitter; data written here is, in that algorithm, renamed into place. Value: "_temporary".

See Also:

Constant Field Values

TEMP_DATA

public static final String TEMP_DATA

Temp data which is not auto-committed: "__temp-data". Uses a different name from normal just to make clear it is different.

See Also:

Constant Field Values

CREATE_SUCCESSFUL_JOB_OUTPUT_DIR_MARKER

public static final String CREATE_SUCCESSFUL_JOB_OUTPUT_DIR_MARKER

Flag to trigger creation of a marker file on job completion.

See Also:

Constant Field Values

_SUCCESS

public static final String _SUCCESS

Marker file to create on success: "_SUCCESS".

See Also:

Constant Field Values

DEFAULT_CREATE_SUCCESSFUL_JOB_DIR_MARKER

public static final boolean DEFAULT_CREATE_SUCCESSFUL_JOB_DIR_MARKER

Default job marker option: true.

See Also:

Constant Field Values

S3A_COMMITTER_FACTORY_KEY

public static final String S3A_COMMITTER_FACTORY_KEY

Key to set for the S3A schema to use the specific committer.

S3A_COMMITTER_FACTORY

public static final String S3A_COMMITTER_FACTORY

S3 Committer factory: "org.apache.hadoop.fs.s3a.commit.S3ACommitterFactory". This uses the value of FS_S3A_COMMITTER_NAME to choose the final committer.

See Also:

Constant Field Values

FS_S3A_COMMITTER_NAME

public static final String FS_S3A_COMMITTER_NAME

Option to identify the S3A committer: "fs.s3a.committer.name".

See Also:

Constant Field Values

COMMITTER_NAME_FILE

public static final String COMMITTER_NAME_FILE

Option for FS_S3A_COMMITTER_NAME: classic/file output committer: "file".

See Also:

Constant Field Values

COMMITTER_NAME_MAGIC

public static final String COMMITTER_NAME_MAGIC

Option for FS_S3A_COMMITTER_NAME: magic output committer: "magic".

See Also:

Constant Field Values

COMMITTER_NAME_DIRECTORY

public static final String COMMITTER_NAME_DIRECTORY

Option for FS_S3A_COMMITTER_NAME: directory output committer: "directory".

See Also:

Constant Field Values

COMMITTER_NAME_PARTITIONED

public static final String COMMITTER_NAME_PARTITIONED

Option for FS_S3A_COMMITTER_NAME: partition output committer: "partitioned".

See Also:

Constant Field Values

FS_S3A_COMMITTER_STAGING_UNIQUE_FILENAMES

public static final String FS_S3A_COMMITTER_STAGING_UNIQUE_FILENAMES

Option for final files to have a uniqueness name through job attempt info, falling back to a new UUID if there is no job attempt information to use. "fs.s3a.committer.staging.unique-filenames". When writing data with the "append" conflict option, this guarantees that new data will not overwrite any existing data.

See Also:

Constant Field Values

DEFAULT_STAGING_COMMITTER_UNIQUE_FILENAMES

public static final boolean DEFAULT_STAGING_COMMITTER_UNIQUE_FILENAMES

Default value for FS_S3A_COMMITTER_STAGING_UNIQUE_FILENAMES: true.

See Also:

Constant Field Values

FS_S3A_COMMITTER_STAGING_CONFLICT_MODE

public static final String FS_S3A_COMMITTER_STAGING_CONFLICT_MODE

Staging committer conflict resolution policy: "fs.s3a.committer.staging.conflict-mode". Supported: fail, append, replace.

See Also:

Constant Field Values

CONFLICT_MODE_FAIL

public static final String CONFLICT_MODE_FAIL

Conflict mode: "fail".

See Also:

Constant Field Values

CONFLICT_MODE_APPEND

public static final String CONFLICT_MODE_APPEND

Conflict mode: "append".

See Also:

Constant Field Values

CONFLICT_MODE_REPLACE

public static final String CONFLICT_MODE_REPLACE

Conflict mode: "replace".

See Also:

Constant Field Values

DEFAULT_CONFLICT_MODE

public static final String DEFAULT_CONFLICT_MODE

Default conflict mode: "fail".

See Also:

Constant Field Values

FS_S3A_COMMITTER_THREADS

public static final String FS_S3A_COMMITTER_THREADS

Number of threads in committers for parallel operations on files (upload, commit, abort, delete...): "fs.s3a.committer.threads".

See Also:

Constant Field Values

DEFAULT_COMMITTER_THREADS

public static final int DEFAULT_COMMITTER_THREADS

Default value for FS_S3A_COMMITTER_THREADS: 8.

See Also:

Constant Field Values

FS_S3A_COMMITTER_STAGING_TMP_PATH

public static final String FS_S3A_COMMITTER_STAGING_TMP_PATH

Path in the cluster filesystem for temporary data: "fs.s3a.committer.staging.tmp.path". This is for HDFS, not the local filesystem. It is only for the summary data of each file, not the actual data being committed.

See Also:

Constant Field Values

FS_S3A_COMMITTER_STAGING_ABORT_PENDING_UPLOADS

public static final String FS_S3A_COMMITTER_STAGING_ABORT_PENDING_UPLOADS

Should the staging committers abort all pending uploads to the destination directory? Default: true. Changing this is if more than one partitioned committer is writing to the same destination tree simultaneously; otherwise the first job to complete will cancel all outstanding uploads from the others. However, it may lead to leaked outstanding uploads from failed tasks. If disabled, configure the bucket lifecycle to remove uploads after a time period, and/or set up a workflow to explicitly delete entries. Otherwise there is a risk that uncommitted uploads may run up bills.

See Also:

Constant Field Values