CommitConstants (Apache Hadoop Amazon Web Services support 3.3.2 API)

java.lang.Object
- org.apache.hadoop.fs.s3a.commit.CommitConstants

@InterfaceAudience.Public
 @InterfaceStability.Unstable
public final class CommitConstants
extends Object

Constants for working with committers.

Field Summary

Fields
Modifier and Type	Field and Description
`static String`	`_SUCCESS` Marker file to create on success: "_SUCCESS".
`static String`	`BASE` Marker of the start of a directory tree for calculating the final path names: "__base".
`static String`	`COMMITTER_NAME_DIRECTORY` Option for `FS_S3A_COMMITTER_NAME`: directory output committer: "directory".
`static String`	`COMMITTER_NAME_FILE` Option for `FS_S3A_COMMITTER_NAME`: classic/file output committer: "file".
`static String`	`COMMITTER_NAME_MAGIC` Option for `FS_S3A_COMMITTER_NAME`: magic output committer: "magic".
`static String`	`COMMITTER_NAME_PARTITIONED` Option for `FS_S3A_COMMITTER_NAME`: partition output committer: "partitioned".
`static String`	`CONFLICT_MODE_APPEND` Conflict mode: "append".
`static String`	`CONFLICT_MODE_FAIL` Conflict mode: "fail".
`static String`	`CONFLICT_MODE_REPLACE` Conflict mode: "replace".
`static String`	`CREATE_SUCCESSFUL_JOB_OUTPUT_DIR_MARKER` Flag to trigger creation of a marker file on job completion.
`static int`	`DEFAULT_COMMITTER_THREADS` Default value for `FS_S3A_COMMITTER_THREADS`: 8.
`static String`	`DEFAULT_CONFLICT_MODE` Default conflict mode: "append".
`static boolean`	`DEFAULT_CREATE_SUCCESSFUL_JOB_DIR_MARKER` Default job marker option: true.
`static boolean`	`DEFAULT_FS_S3A_COMMITTER_ABORT_PENDING_UPLOADS` Default configuration value for `FS_S3A_COMMITTER_ABORT_PENDING_UPLOADS`.
`static boolean`	`DEFAULT_MAGIC_COMMITTER_ENABLED` Is the committer enabled by default: true.
`static boolean`	`DEFAULT_S3A_COMMITTER_GENERATE_UUID` Default value for `FS_S3A_COMMITTER_GENERATE_UUID`: false.
`static boolean`	`DEFAULT_S3A_COMMITTER_REQUIRE_UUID` Default value for `FS_S3A_COMMITTER_REQUIRE_UUID`: false.
`static boolean`	`DEFAULT_STAGING_COMMITTER_UNIQUE_FILENAMES` Default value for `FS_S3A_COMMITTER_STAGING_UNIQUE_FILENAMES`: true.
`static String`	`FS_S3A_COMMITTER_ABORT_PENDING_UPLOADS` Should committers abort all pending uploads to the destination directory?
`static String`	`FS_S3A_COMMITTER_GENERATE_UUID` Generate a UUID in job setup rather than fall back to YARN Application attempt ID.
`static String`	`FS_S3A_COMMITTER_NAME` Option to identify the S3A committer: "fs.s3a.committer.name".
`static String`	`FS_S3A_COMMITTER_REQUIRE_UUID` Require the spark UUID to be passed down: "fs.s3a.committer.require.uuid".
`static String`	`FS_S3A_COMMITTER_STAGING_ABORT_PENDING_UPLOADS` Deprecated.
`static String`	`FS_S3A_COMMITTER_STAGING_CONFLICT_MODE` Staging committer conflict resolution policy: "fs.s3a.committer.staging.conflict-mode".
`static String`	`FS_S3A_COMMITTER_STAGING_TMP_PATH` Path in the cluster filesystem for temporary data: "fs.s3a.committer.staging.tmp.path".
`static String`	`FS_S3A_COMMITTER_STAGING_UNIQUE_FILENAMES` Option for final files to have a uniqueness name through job attempt info, falling back to a new UUID if there is no job attempt information to use.
`static String`	`FS_S3A_COMMITTER_THREADS` Number of threads in committers for parallel operations on files (upload, commit, abort, delete...): "fs.s3a.committer.threads".
`static String`	`MAGIC` Path for "magic" writes: path and `PENDING_SUFFIX` files: "__magic".
`static String`	`MAGIC_COMMITTER_ENABLED` Flag to indicate whether support for the Magic committer is enabled in the filesystem.
`static String`	`MAGIC_COMMITTER_PREFIX` Flag to indicate whether support for the Magic committer is enabled in the filesystem.
`static String`	`PENDING_SUFFIX` Suffix applied to pending commit metadata: ".pending".
`static String`	`PENDINGSET_SUFFIX` Suffix applied to multiple pending commit metadata: ".pendingset".
`static String`	`S3A_COMMITTER_FACTORY` S3 Committer factory: "org.apache.hadoop.fs.s3a.commit.S3ACommitterFactory".
`static String`	`S3A_COMMITTER_FACTORY_KEY` Key to set for the S3A schema to use the specific committer.
`static String`	`STORE_CAPABILITY_MAGIC_COMMITTER` Flag to indicate that a store supports magic committers.
`static String`	`STORE_CAPABILITY_MAGIC_COMMITTER_OLD` Deprecated.
`static String`	`STREAM_CAPABILITY_MAGIC_OUTPUT` Flag to indicate whether a stream is a magic output stream; returned in `StreamCapabilities` Value: "fs.s3a.capability.magic.output.stream".
`static String`	`STREAM_CAPABILITY_MAGIC_OUTPUT_OLD` Deprecated.
`static int`	`SUCCESS_MARKER_FILE_LIMIT` The limit to the number of committed objects tracked during job commits and saved to the _SUCCESS file.
`static String`	`TASK_ATTEMPT_ID` Extra Data key for task attempt in pendingset files.
`static String`	`TEMP_DATA` Temp data which is not auto-committed: "__temp-data".
`static String`	`TEMPORARY` This is the "Pending" directory of the `FileOutputCommitter`; data written here is, in that algorithm, renamed into place.
`static String`	`X_HEADER_MAGIC_MARKER` Magic Marker header to declare final file length on magic uploads marker objects: "x-hadoop-s3a-magic-data-length".
`static String`	`XA_MAGIC_MARKER` XAttr name of magic marker, with "header." prefix: "header.x-hadoop-s3a-magic-data-length".

Method Summary
- Methods inherited from class java.lang.Object
  clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Field Detail
  - MAGIC
```
public static final String MAGIC
```
    Path for "magic" writes: path and PENDING_SUFFIX files: "__magic".
    
    See Also:
    
    Constant Field Values
  - BASE
```
public static final String BASE
```
    Marker of the start of a directory tree for calculating the final path names: "__base".
    
    See Also:
    
    Constant Field Values
  - PENDING_SUFFIX
```
public static final String PENDING_SUFFIX
```
    Suffix applied to pending commit metadata: ".pending".
    
    See Also:
    
    Constant Field Values
  - PENDINGSET_SUFFIX
```
public static final String PENDINGSET_SUFFIX
```
    Suffix applied to multiple pending commit metadata: ".pendingset".
    
    See Also:
    
    Constant Field Values
  - MAGIC_COMMITTER_PREFIX
```
public static final String MAGIC_COMMITTER_PREFIX
```
    Flag to indicate whether support for the Magic committer is enabled in the filesystem. Value: "fs.s3a.committer.magic".
    
    See Also:
    
    Constant Field Values
  - MAGIC_COMMITTER_ENABLED
```
public static final String MAGIC_COMMITTER_ENABLED
```
    Flag to indicate whether support for the Magic committer is enabled in the filesystem. Value: "fs.s3a.committer.magic.enabled".
    
    See Also:
    
    Constant Field Values
  - STREAM_CAPABILITY_MAGIC_OUTPUT
```
public static final String STREAM_CAPABILITY_MAGIC_OUTPUT
```
    Flag to indicate whether a stream is a magic output stream; returned in StreamCapabilities Value: "fs.s3a.capability.magic.output.stream".
    
    See Also:
    
    Constant Field Values
  - STORE_CAPABILITY_MAGIC_COMMITTER
```
public static final String STORE_CAPABILITY_MAGIC_COMMITTER
```
    Flag to indicate that a store supports magic committers. returned in PathCapabilities Value: "fs.s3a.capability.magic.committer".
    
    See Also:
    
    Constant Field Values
  - STREAM_CAPABILITY_MAGIC_OUTPUT_OLD
```
@Deprecated
public static final String STREAM_CAPABILITY_MAGIC_OUTPUT_OLD
```
    Deprecated.
    
    Flag to indicate whether a stream is a magic output stream; returned in StreamCapabilities Value: "s3a:magic.output.stream".
    
    See Also:
    
    Constant Field Values
  - STORE_CAPABILITY_MAGIC_COMMITTER_OLD
```
@Deprecated
public static final String STORE_CAPABILITY_MAGIC_COMMITTER_OLD
```
    Deprecated.
    
    Flag to indicate that a store supports magic committers. returned in PathCapabilities Value: "s3a:magic.committer".
    
    See Also:
    
    Constant Field Values
  - DEFAULT_MAGIC_COMMITTER_ENABLED
```
public static final boolean DEFAULT_MAGIC_COMMITTER_ENABLED
```
    Is the committer enabled by default: true.
    
    See Also:
    
    Constant Field Values
  - TEMPORARY
```
public static final String TEMPORARY
```
    This is the "Pending" directory of the FileOutputCommitter; data written here is, in that algorithm, renamed into place. Value: "_temporary".
    
    See Also:
    
    Constant Field Values
  - TEMP_DATA
```
public static final String TEMP_DATA
```
    Temp data which is not auto-committed: "__temp-data". Uses a different name from normal just to make clear it is different.
    
    See Also:
    
    Constant Field Values
  - CREATE_SUCCESSFUL_JOB_OUTPUT_DIR_MARKER
```
public static final String CREATE_SUCCESSFUL_JOB_OUTPUT_DIR_MARKER
```
    Flag to trigger creation of a marker file on job completion.
    
    See Also:
    
    Constant Field Values
  - _SUCCESS
```
public static final String _SUCCESS
```
    Marker file to create on success: "_SUCCESS".
    
    See Also:
    
    Constant Field Values
  - DEFAULT_CREATE_SUCCESSFUL_JOB_DIR_MARKER
```
public static final boolean DEFAULT_CREATE_SUCCESSFUL_JOB_DIR_MARKER
```
    Default job marker option: true.
    
    See Also:
    
    Constant Field Values
  - S3A_COMMITTER_FACTORY_KEY
```
public static final String S3A_COMMITTER_FACTORY_KEY
```
    Key to set for the S3A schema to use the specific committer.
  - S3A_COMMITTER_FACTORY
```
public static final String S3A_COMMITTER_FACTORY
```
    S3 Committer factory: "org.apache.hadoop.fs.s3a.commit.S3ACommitterFactory". This uses the value of FS_S3A_COMMITTER_NAME to choose the final committer.
    
    See Also:
    
    Constant Field Values
  - FS_S3A_COMMITTER_NAME
```
public static final String FS_S3A_COMMITTER_NAME
```
    Option to identify the S3A committer: "fs.s3a.committer.name".
    
    See Also:
    
    Constant Field Values
  - COMMITTER_NAME_FILE
```
public static final String COMMITTER_NAME_FILE
```
    Option for FS_S3A_COMMITTER_NAME: classic/file output committer: "file".
    
    See Also:
    
    Constant Field Values
  - COMMITTER_NAME_MAGIC
```
public static final String COMMITTER_NAME_MAGIC
```
    Option for FS_S3A_COMMITTER_NAME: magic output committer: "magic".
    
    See Also:
    
    Constant Field Values
  - COMMITTER_NAME_DIRECTORY
```
public static final String COMMITTER_NAME_DIRECTORY
```
    Option for FS_S3A_COMMITTER_NAME: directory output committer: "directory".
    
    See Also:
    
    Constant Field Values
  - COMMITTER_NAME_PARTITIONED
```
public static final String COMMITTER_NAME_PARTITIONED
```
    Option for FS_S3A_COMMITTER_NAME: partition output committer: "partitioned".
    
    See Also:
    
    Constant Field Values
  - FS_S3A_COMMITTER_STAGING_UNIQUE_FILENAMES
```
public static final String FS_S3A_COMMITTER_STAGING_UNIQUE_FILENAMES
```
    Option for final files to have a uniqueness name through job attempt info, falling back to a new UUID if there is no job attempt information to use. "fs.s3a.committer.staging.unique-filenames". When writing data with the "append" conflict option, this guarantees that new data will not overwrite any existing data.
    
    See Also:
    
    Constant Field Values
  - DEFAULT_STAGING_COMMITTER_UNIQUE_FILENAMES
```
public static final boolean DEFAULT_STAGING_COMMITTER_UNIQUE_FILENAMES
```
    Default value for FS_S3A_COMMITTER_STAGING_UNIQUE_FILENAMES: true.
    
    See Also:
    
    Constant Field Values
  - FS_S3A_COMMITTER_STAGING_CONFLICT_MODE
```
public static final String FS_S3A_COMMITTER_STAGING_CONFLICT_MODE
```
    Staging committer conflict resolution policy: "fs.s3a.committer.staging.conflict-mode". Supported: fail, append, replace.
    
    See Also:
    
    Constant Field Values
  - CONFLICT_MODE_FAIL
```
public static final String CONFLICT_MODE_FAIL
```
    Conflict mode: "fail".
    
    See Also:
    
    Constant Field Values
  - CONFLICT_MODE_APPEND
```
public static final String CONFLICT_MODE_APPEND
```
    Conflict mode: "append".
    
    See Also:
    
    Constant Field Values
  - CONFLICT_MODE_REPLACE
```
public static final String CONFLICT_MODE_REPLACE
```
    Conflict mode: "replace".
    
    See Also:
    
    Constant Field Values
  - DEFAULT_CONFLICT_MODE
```
public static final String DEFAULT_CONFLICT_MODE
```
    Default conflict mode: "append".
    
    See Also:
    
    Constant Field Values
  - FS_S3A_COMMITTER_THREADS
```
public static final String FS_S3A_COMMITTER_THREADS
```
    Number of threads in committers for parallel operations on files (upload, commit, abort, delete...): "fs.s3a.committer.threads".
    
    See Also:
    
    Constant Field Values
  - DEFAULT_COMMITTER_THREADS
```
public static final int DEFAULT_COMMITTER_THREADS
```
    Default value for FS_S3A_COMMITTER_THREADS: 8.
    
    See Also:
    
    Constant Field Values
  - FS_S3A_COMMITTER_STAGING_TMP_PATH
```
public static final String FS_S3A_COMMITTER_STAGING_TMP_PATH
```
    Path in the cluster filesystem for temporary data: "fs.s3a.committer.staging.tmp.path". This is for HDFS, not the local filesystem. It is only for the summary data of each file, not the actual data being committed.
    
    See Also:
    
    Constant Field Values
  - FS_S3A_COMMITTER_STAGING_ABORT_PENDING_UPLOADS
```
@Deprecated
public static final String FS_S3A_COMMITTER_STAGING_ABORT_PENDING_UPLOADS
```
    Deprecated.
    
    Should committers abort all pending uploads to the destination directory?
    Deprecated: switch to FS_S3A_COMMITTER_ABORT_PENDING_UPLOADS.
    
    See Also:
    
    Constant Field Values
  - FS_S3A_COMMITTER_ABORT_PENDING_UPLOADS
```
public static final String FS_S3A_COMMITTER_ABORT_PENDING_UPLOADS
```
    Should committers abort all pending uploads to the destination directory?
    Value: "fs.s3a.committer.abort.pending.uploads".
    Change this is if more than one committer is writing to the same destination tree simultaneously; otherwise the first job to complete will cancel all outstanding uploads from the others. If disabled, configure the bucket lifecycle to remove uploads after a time period, and/or set up a workflow to explicitly delete entries. Otherwise there is a risk that uncommitted uploads may run up bills.
    
    See Also:
    
    Constant Field Values
  - DEFAULT_FS_S3A_COMMITTER_ABORT_PENDING_UPLOADS
```
public static final boolean DEFAULT_FS_S3A_COMMITTER_ABORT_PENDING_UPLOADS
```
    Default configuration value for FS_S3A_COMMITTER_ABORT_PENDING_UPLOADS. Value: true.
    
    See Also:
    
    Constant Field Values
  - SUCCESS_MARKER_FILE_LIMIT
```
public static final int SUCCESS_MARKER_FILE_LIMIT
```
    The limit to the number of committed objects tracked during job commits and saved to the _SUCCESS file.
    
    See Also:
    
    Constant Field Values
  - TASK_ATTEMPT_ID
```
public static final String TASK_ATTEMPT_ID
```
    Extra Data key for task attempt in pendingset files.
    
    See Also:
    
    Constant Field Values
  - FS_S3A_COMMITTER_REQUIRE_UUID
```
public static final String FS_S3A_COMMITTER_REQUIRE_UUID
```
    Require the spark UUID to be passed down: "fs.s3a.committer.require.uuid". This is to verify that SPARK-33230 has been applied to spark, and that InternalCommitterConstants.SPARK_WRITE_UUID is set.
    MUST ONLY BE SET WITH SPARK JOBS.
    
    See Also:
    
    Constant Field Values
  - DEFAULT_S3A_COMMITTER_REQUIRE_UUID
```
public static final boolean DEFAULT_S3A_COMMITTER_REQUIRE_UUID
```
    Default value for FS_S3A_COMMITTER_REQUIRE_UUID: false.
    
    See Also:
    
    Constant Field Values
  - FS_S3A_COMMITTER_GENERATE_UUID
```
public static final String FS_S3A_COMMITTER_GENERATE_UUID
```
    Generate a UUID in job setup rather than fall back to YARN Application attempt ID.
    MUST ONLY BE SET WITH SPARK JOBS.
    
    See Also:
    
    Constant Field Values
  - DEFAULT_S3A_COMMITTER_GENERATE_UUID
```
public static final boolean DEFAULT_S3A_COMMITTER_GENERATE_UUID
```
    Default value for FS_S3A_COMMITTER_GENERATE_UUID: false.
    
    See Also:
    
    Constant Field Values
  - X_HEADER_MAGIC_MARKER
```
public static final String X_HEADER_MAGIC_MARKER
```
    Magic Marker header to declare final file length on magic uploads marker objects: "x-hadoop-s3a-magic-data-length".
    
    See Also:
    
    Constant Field Values
  - XA_MAGIC_MARKER
```
public static final String XA_MAGIC_MARKER
```
    XAttr name of magic marker, with "header." prefix: "header.x-hadoop-s3a-magic-data-length".
    
    See Also:
    
    Constant Field Values

Class CommitConstants

Field Summary

Method Summary

Methods inherited from class java.lang.Object

Field Detail

MAGIC

BASE

PENDING_SUFFIX

PENDINGSET_SUFFIX

MAGIC_COMMITTER_PREFIX

MAGIC_COMMITTER_ENABLED

STREAM_CAPABILITY_MAGIC_OUTPUT

STORE_CAPABILITY_MAGIC_COMMITTER

STREAM_CAPABILITY_MAGIC_OUTPUT_OLD

STORE_CAPABILITY_MAGIC_COMMITTER_OLD

DEFAULT_MAGIC_COMMITTER_ENABLED

TEMPORARY

TEMP_DATA

CREATE_SUCCESSFUL_JOB_OUTPUT_DIR_MARKER

_SUCCESS

DEFAULT_CREATE_SUCCESSFUL_JOB_DIR_MARKER

S3A_COMMITTER_FACTORY_KEY

S3A_COMMITTER_FACTORY

FS_S3A_COMMITTER_NAME

COMMITTER_NAME_FILE

COMMITTER_NAME_MAGIC

COMMITTER_NAME_DIRECTORY

COMMITTER_NAME_PARTITIONED

FS_S3A_COMMITTER_STAGING_UNIQUE_FILENAMES

DEFAULT_STAGING_COMMITTER_UNIQUE_FILENAMES

FS_S3A_COMMITTER_STAGING_CONFLICT_MODE

CONFLICT_MODE_FAIL

CONFLICT_MODE_APPEND

CONFLICT_MODE_REPLACE

DEFAULT_CONFLICT_MODE

FS_S3A_COMMITTER_THREADS

DEFAULT_COMMITTER_THREADS

FS_S3A_COMMITTER_STAGING_TMP_PATH

FS_S3A_COMMITTER_STAGING_ABORT_PENDING_UPLOADS

FS_S3A_COMMITTER_ABORT_PENDING_UPLOADS

DEFAULT_FS_S3A_COMMITTER_ABORT_PENDING_UPLOADS

SUCCESS_MARKER_FILE_LIMIT

TASK_ATTEMPT_ID

FS_S3A_COMMITTER_REQUIRE_UUID

DEFAULT_S3A_COMMITTER_REQUIRE_UUID

FS_S3A_COMMITTER_GENERATE_UUID

DEFAULT_S3A_COMMITTER_GENERATE_UUID

X_HEADER_MAGIC_MARKER

XA_MAGIC_MARKER