org.apache.hadoop.fs.s3a.commit.impl

Class CommitUtilsWithMR

java.lang.Object

org.apache.hadoop.fs.s3a.commit.impl.CommitUtilsWithMR

@InterfaceAudience.Private
 @InterfaceStability.Unstable
public final class CommitUtilsWithMR
extends Object

These are commit utility methods which import classes from hadoop-mapreduce, and so only work when that module is on the classpath. Do not use in any codepath intended to be used from the S3AFS except in the committers themselves.

Method Summary

Modifier and Type	Method and Description
static String	formatAppAttemptDir(String jobUUID, int appAttemptId) Build the name of the job attempt directory.
static String	formatJobDir(String jobUUID) Build the name of the job directory, without app attempt.
static int	getAppAttemptId(org.apache.hadoop.mapreduce.JobContext context) Get the Application Attempt ID for this job.
static org.apache.hadoop.fs.Path	getBaseMagicTaskAttemptPath(org.apache.hadoop.mapreduce.TaskAttemptContext context, String jobUUID, org.apache.hadoop.fs.Path dest) Get the base Magic attempt path, without any annotations to mark relative references.
static String	getConfigurationOption(org.apache.hadoop.mapreduce.JobContext context, org.apache.hadoop.conf.Configuration fsConf, String key, String defVal) Get a configuration option, with any value in the job configuration taking priority over that in the filesystem.
static org.apache.hadoop.fs.Path	getMagicJobAttemptPath(String jobUUID, int appAttemptId, org.apache.hadoop.fs.Path dest) Compute the "magic" path for a job attempt.
static org.apache.hadoop.fs.Path	getMagicJobAttemptsPath(org.apache.hadoop.fs.Path out, String jobUUID) Get the location of magic job attempts.
static org.apache.hadoop.fs.Path	getMagicJobPath(String jobUUID, org.apache.hadoop.fs.Path dest) Compute the "magic" path for a job.
static org.apache.hadoop.fs.Path	getMagicTaskAttemptPath(org.apache.hadoop.mapreduce.TaskAttemptContext context, String jobUUID, org.apache.hadoop.fs.Path dest) Compute the path where the output of a task attempt is stored until that task is committed.
static org.apache.hadoop.fs.Path	getMagicTaskAttemptsPath(String jobUUID, org.apache.hadoop.fs.Path dest, int appAttemptId) Compute the path where the output of magic task attempts are stored.
static org.apache.hadoop.fs.Path	getTempJobAttemptPath(String jobUUID, org.apache.hadoop.fs.Path out, int appAttemptId) Compute a path for temporary data associated with a job.
static org.apache.hadoop.fs.Path	getTempTaskAttemptPath(org.apache.hadoop.mapreduce.TaskAttemptContext context, String jobUUID, org.apache.hadoop.fs.Path out) Compute the path where the output of a given task attempt will be placed.
static String	jobIdString(org.apache.hadoop.mapreduce.JobContext context) Get a string value of a job ID; returns meaningful text if there is no ID.
static String	jobName(org.apache.hadoop.mapreduce.JobContext context) Get a job name; returns meaningful text if there is no name.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

getMagicJobAttemptsPath

public static org.apache.hadoop.fs.Path getMagicJobAttemptsPath(org.apache.hadoop.fs.Path out,
                                                                String jobUUID)

Get the location of magic job attempts.

Parameters:

out - the base output directory.

jobUUID - unique Job ID.

Returns:

the location of magic job attempts.

getAppAttemptId

public static int getAppAttemptId(org.apache.hadoop.mapreduce.JobContext context)

Get the Application Attempt ID for this job.

Parameters:

context - the context to look in

Returns:

the Application Attempt ID for a given job, or 0

getMagicJobAttemptPath

public static org.apache.hadoop.fs.Path getMagicJobAttemptPath(String jobUUID,
                                                               int appAttemptId,
                                                               org.apache.hadoop.fs.Path dest)

Compute the "magic" path for a job attempt.

Parameters:

jobUUID - unique Job ID.

appAttemptId - the ID of the application attempt for this job.

dest - the final output directory

Returns:

the path to store job attempt data.

getMagicJobPath

public static org.apache.hadoop.fs.Path getMagicJobPath(String jobUUID,
                                                        org.apache.hadoop.fs.Path dest)

Compute the "magic" path for a job.

Parameters:

jobUUID - unique Job ID.

dest - the final output directory

Returns:

the path to store job attempt data.

formatJobDir

public static String formatJobDir(String jobUUID)

Build the name of the job directory, without app attempt. This is the path to use for cleanup.

Parameters:

jobUUID - unique Job ID.

Returns:

the directory name for the job

formatAppAttemptDir

public static String formatAppAttemptDir(String jobUUID,
                                         int appAttemptId)

Build the name of the job attempt directory.

Parameters:

jobUUID - unique Job ID.

appAttemptId - the ID of the application attempt for this job.

Returns:

the directory tree for the application attempt

getMagicTaskAttemptsPath

public static org.apache.hadoop.fs.Path getMagicTaskAttemptsPath(String jobUUID,
                                                                 org.apache.hadoop.fs.Path dest,
                                                                 int appAttemptId)

Compute the path where the output of magic task attempts are stored.

Parameters:

jobUUID - unique Job ID.

dest - destination of work

appAttemptId - the ID of the application attempt for this job.

Returns:

the path where the output of magic task attempts are stored.

getMagicTaskAttemptPath

public static org.apache.hadoop.fs.Path getMagicTaskAttemptPath(org.apache.hadoop.mapreduce.TaskAttemptContext context,
                                                                String jobUUID,
                                                                org.apache.hadoop.fs.Path dest)

Compute the path where the output of a task attempt is stored until that task is committed. This path is marked as a base path for relocations, so subdirectory information is preserved.

Parameters:

context - the context of the task attempt.

jobUUID - unique Job ID.

dest - The output path to commit work into

Returns:

the path where a task attempt should be stored.

getBaseMagicTaskAttemptPath

public static org.apache.hadoop.fs.Path getBaseMagicTaskAttemptPath(org.apache.hadoop.mapreduce.TaskAttemptContext context,
                                                                    String jobUUID,
                                                                    org.apache.hadoop.fs.Path dest)

Get the base Magic attempt path, without any annotations to mark relative references. If there is an app attempt property in the context configuration, that is included.

Parameters:

context - task context.

jobUUID - unique Job ID.

dest - The output path to commit work into

Returns:

the path under which all attempts go

getTempJobAttemptPath

public static org.apache.hadoop.fs.Path getTempJobAttemptPath(String jobUUID,
                                                              org.apache.hadoop.fs.Path out,
                                                              int appAttemptId)

Compute a path for temporary data associated with a job. This data is not magic

Parameters:

jobUUID - unique Job ID.

out - output directory of job

appAttemptId - the ID of the application attempt for this job.

Returns:

the path to store temporary job attempt data.

getTempTaskAttemptPath

public static org.apache.hadoop.fs.Path getTempTaskAttemptPath(org.apache.hadoop.mapreduce.TaskAttemptContext context,
                                                               String jobUUID,
                                                               org.apache.hadoop.fs.Path out)

Compute the path where the output of a given task attempt will be placed.

Parameters:

context - task context

jobUUID - unique Job ID.

out - output directory of job

Returns:

the path to store temporary job attempt data.

jobIdString

public static String jobIdString(org.apache.hadoop.mapreduce.JobContext context)

Get a string value of a job ID; returns meaningful text if there is no ID.

Parameters:

context - job context

Returns:

a string for logs

jobName

public static String jobName(org.apache.hadoop.mapreduce.JobContext context)

Get a job name; returns meaningful text if there is no name.

Parameters:

context - job context

Returns:

a string for logs

getConfigurationOption

public static String getConfigurationOption(org.apache.hadoop.mapreduce.JobContext context,
                                            org.apache.hadoop.conf.Configuration fsConf,
                                            String key,
                                            String defVal)

Get a configuration option, with any value in the job configuration taking priority over that in the filesystem. This allows for per-job override of FS parameters. Order is: job context, filesystem config, default value

Parameters:

context - job/task context

fsConf - filesystem configuration. Get this from the FS to guarantee per-bucket parameter propagation

key - key to look for

defVal - default value

Returns:

the configuration option.