org.apache.hadoop.fs.s3a.commit

Class CommitUtilsWithMR

java.lang.Object

org.apache.hadoop.fs.s3a.commit.CommitUtilsWithMR

@InterfaceAudience.Private
 @InterfaceStability.Unstable
public final class CommitUtilsWithMR
extends Object

These are commit utility methods which import classes from hadoop-mapreduce, and so only work when that module is on the classpath. Do not use in any codepath intended to be used from the S3AFS except in the committers themselves.

Method Summary

Modifier and Type	Method and Description
static String	formatAppAttemptDir(int attemptId) Format the application attempt directory.
static int	getAppAttemptId(org.apache.hadoop.mapreduce.JobContext context) Get the Application Attempt ID for this job.
static org.apache.hadoop.fs.Path	getBaseMagicTaskAttemptPath(org.apache.hadoop.mapreduce.TaskAttemptContext context, org.apache.hadoop.fs.Path dest) Get the base Magic attempt path, without any annotations to mark relative references.
static String	getConfigurationOption(org.apache.hadoop.mapreduce.JobContext context, org.apache.hadoop.conf.Configuration fsConf, String key, String defVal) Get a configuration option, with any value in the job configuration taking priority over that in the filesystem.
static org.apache.hadoop.fs.Path	getMagicJobAttemptPath(int appAttemptId, org.apache.hadoop.fs.Path dest) Compute the "magic" path for a job attempt.
static org.apache.hadoop.fs.Path	getMagicJobAttemptsPath(org.apache.hadoop.fs.Path out) Get the location of magic job attempts.
static org.apache.hadoop.fs.Path	getMagicTaskAttemptPath(org.apache.hadoop.mapreduce.TaskAttemptContext context, org.apache.hadoop.fs.Path dest) Compute the path where the output of a task attempt is stored until that task is committed.
static org.apache.hadoop.fs.Path	getMagicTaskAttemptsPath(org.apache.hadoop.mapreduce.JobContext context, org.apache.hadoop.fs.Path dest) Compute the path where the output of magic task attempts are stored.
static org.apache.hadoop.fs.Path	getTempJobAttemptPath(int appAttemptId, org.apache.hadoop.fs.Path out) Compute a path for temporary data associated with a job.
static org.apache.hadoop.fs.Path	getTempTaskAttemptPath(org.apache.hadoop.mapreduce.TaskAttemptContext context, org.apache.hadoop.fs.Path out) Compute the path where the output of a given job attempt will be placed.
static String	jobIdString(org.apache.hadoop.mapreduce.JobContext context) Get a string value of a job ID; returns meaningful text if there is no ID.
static String	jobName(org.apache.hadoop.mapreduce.JobContext context) Get a job name; returns meaningful text if there is no name.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

getMagicJobAttemptsPath

public static org.apache.hadoop.fs.Path getMagicJobAttemptsPath(org.apache.hadoop.fs.Path out)

Get the location of magic job attempts.

Parameters:

out - the base output directory.

Returns:

the location of magic job attempts.

getAppAttemptId

public static int getAppAttemptId(org.apache.hadoop.mapreduce.JobContext context)

Get the Application Attempt ID for this job.

Parameters:

context - the context to look in

Returns:

the Application Attempt ID for a given job.

getMagicJobAttemptPath

public static org.apache.hadoop.fs.Path getMagicJobAttemptPath(int appAttemptId,
                                                               org.apache.hadoop.fs.Path dest)

Compute the "magic" path for a job attempt.

Parameters:

appAttemptId - the ID of the application attempt for this job.

dest - the final output directory

Returns:

the path to store job attempt data.

formatAppAttemptDir

public static String formatAppAttemptDir(int attemptId)

Format the application attempt directory.

Parameters:

attemptId - attempt ID

Returns:

the directory name for the application attempt

getMagicTaskAttemptsPath

public static org.apache.hadoop.fs.Path getMagicTaskAttemptsPath(org.apache.hadoop.mapreduce.JobContext context,
                                                                 org.apache.hadoop.fs.Path dest)

Compute the path where the output of magic task attempts are stored.

Parameters:

context - the context of the job with magic tasks.

dest - destination of work

Returns:

the path where the output of magic task attempts are stored.

getMagicTaskAttemptPath

public static org.apache.hadoop.fs.Path getMagicTaskAttemptPath(org.apache.hadoop.mapreduce.TaskAttemptContext context,
                                                                org.apache.hadoop.fs.Path dest)

Compute the path where the output of a task attempt is stored until that task is committed. This path is marked as a base path for relocations, so subdirectory information is preserved.

Parameters:

context - the context of the task attempt.

dest - The output path to commit work into

Returns:

the path where a task attempt should be stored.

getBaseMagicTaskAttemptPath

public static org.apache.hadoop.fs.Path getBaseMagicTaskAttemptPath(org.apache.hadoop.mapreduce.TaskAttemptContext context,
                                                                    org.apache.hadoop.fs.Path dest)

Get the base Magic attempt path, without any annotations to mark relative references.

Parameters:

context - task context.

dest - The output path to commit work into

Returns:

the path under which all attempts go

getTempJobAttemptPath

public static org.apache.hadoop.fs.Path getTempJobAttemptPath(int appAttemptId,
                                                              org.apache.hadoop.fs.Path out)

Compute a path for temporary data associated with a job. This data is not magic

Parameters:

appAttemptId - the ID of the application attempt for this job.

out - output directory of job

Returns:

the path to store temporary job attempt data.

getTempTaskAttemptPath

public static org.apache.hadoop.fs.Path getTempTaskAttemptPath(org.apache.hadoop.mapreduce.TaskAttemptContext context,
                                                               org.apache.hadoop.fs.Path out)

Compute the path where the output of a given job attempt will be placed.

Parameters:

context - task context

out - output directory of job

Returns:

the path to store temporary job attempt data.

jobIdString

public static String jobIdString(org.apache.hadoop.mapreduce.JobContext context)

Get a string value of a job ID; returns meaningful text if there is no ID.

Parameters:

context - job context

Returns:

a string for logs

jobName

public static String jobName(org.apache.hadoop.mapreduce.JobContext context)

Get a job name; returns meaningful text if there is no name.

Parameters:

context - job context

Returns:

a string for logs

getConfigurationOption

public static String getConfigurationOption(org.apache.hadoop.mapreduce.JobContext context,
                                            org.apache.hadoop.conf.Configuration fsConf,
                                            String key,
                                            String defVal)

Get a configuration option, with any value in the job configuration taking priority over that in the filesystem. This allows for per-job override of FS parameters. Order is: job context, filesystem config, default value

Parameters:

context - job/task context

fsConf - filesystem configuration. Get this from the FS to guarantee per-bucket parameter propagation

key - key to look for

defVal - default value

Returns:

the configuration option.