Modifier and Type | Field and Description |
---|---|
protected JobConf |
Task.conf |
protected JobConf |
Task.CombinerRunner.job |
Modifier and Type | Method and Description |
---|---|
JobConf |
TaskAttemptContextImpl.getJobConf() |
JobConf |
TaskAttemptContext.getJobConf() |
JobConf |
ShuffleConsumerPlugin.Context.getJobConf() |
JobConf |
MapOutputCollector.Context.getJobConf() |
JobConf |
JobContextImpl.getJobConf()
Get the job Configuration
|
JobConf |
JobContext.getJobConf()
Get the job Configuration
|
Modifier and Type | Method and Description |
---|---|
static void |
FileInputFormat.addInputPath(JobConf conf,
org.apache.hadoop.fs.Path path)
Add a
Path to the list of inputs for the map-reduce job. |
static void |
FileInputFormat.addInputPaths(JobConf conf,
String commaSeparatedPaths)
Add the given comma separated paths to the list of inputs for
the map-reduce job.
|
void |
SequenceFileAsBinaryOutputFormat.checkOutputSpecs(org.apache.hadoop.fs.FileSystem ignored,
JobConf job) |
void |
OutputFormat.checkOutputSpecs(org.apache.hadoop.fs.FileSystem ignored,
JobConf job)
Check for validity of the output-specification for the job.
|
void |
FileOutputFormat.checkOutputSpecs(org.apache.hadoop.fs.FileSystem ignored,
JobConf job) |
void |
TextInputFormat.configure(JobConf conf) |
void |
MapRunner.configure(JobConf job) |
void |
MapReduceBase.configure(JobConf job)
Default implementation that does nothing.
|
void |
KeyValueTextInputFormat.configure(JobConf conf) |
void |
JobConfigurable.configure(JobConf job)
Initializes a new instance from a
JobConf . |
void |
FixedLengthInputFormat.configure(JobConf conf) |
static <K,V> Task.CombinerRunner<K,V> |
Task.CombinerRunner.create(JobConf job,
TaskAttemptID taskId,
Counters.Counter inputCounter,
Task.TaskReporter reporter,
OutputCommitter committer) |
static boolean |
FileOutputFormat.getCompressOutput(JobConf conf)
Is the job output compressed?
|
static org.apache.hadoop.fs.PathFilter |
FileInputFormat.getInputPathFilter(JobConf conf)
Get a PathFilter instance of the filter set for the input paths.
|
static org.apache.hadoop.fs.Path[] |
FileInputFormat.getInputPaths(JobConf conf)
Get the list of input
Path s for the map-reduce job. |
static org.apache.hadoop.io.SequenceFile.CompressionType |
SequenceFileOutputFormat.getOutputCompressionType(JobConf conf)
Get the
SequenceFile.CompressionType for the output SequenceFile . |
static Class<? extends org.apache.hadoop.io.compress.CompressionCodec> |
FileOutputFormat.getOutputCompressorClass(JobConf conf,
Class<? extends org.apache.hadoop.io.compress.CompressionCodec> defaultValue)
Get the
CompressionCodec for compressing the job outputs. |
static org.apache.hadoop.fs.Path |
FileOutputFormat.getOutputPath(JobConf conf)
Get the
Path to the output directory for the map-reduce job. |
static org.apache.hadoop.fs.Path |
FileOutputFormat.getPathForCustomFile(JobConf conf,
String name)
Helper function to generate a
Path for a file that is unique for
the task within the job output directory. |
RecordReader<org.apache.hadoop.io.LongWritable,org.apache.hadoop.io.Text> |
TextInputFormat.getRecordReader(InputSplit genericSplit,
JobConf job,
Reporter reporter) |
RecordReader<K,V> |
SequenceFileInputFormat.getRecordReader(InputSplit split,
JobConf job,
Reporter reporter) |
RecordReader<K,V> |
SequenceFileInputFilter.getRecordReader(InputSplit split,
JobConf job,
Reporter reporter)
Create a record reader for the given split
|
RecordReader<org.apache.hadoop.io.Text,org.apache.hadoop.io.Text> |
SequenceFileAsTextInputFormat.getRecordReader(InputSplit split,
JobConf job,
Reporter reporter) |
RecordReader<org.apache.hadoop.io.BytesWritable,org.apache.hadoop.io.BytesWritable> |
SequenceFileAsBinaryInputFormat.getRecordReader(InputSplit split,
JobConf job,
Reporter reporter) |
abstract RecordReader<K,V> |
MultiFileInputFormat.getRecordReader(InputSplit split,
JobConf job,
Reporter reporter) |
RecordReader<org.apache.hadoop.io.Text,org.apache.hadoop.io.Text> |
KeyValueTextInputFormat.getRecordReader(InputSplit genericSplit,
JobConf job,
Reporter reporter) |
RecordReader<K,V> |
InputFormat.getRecordReader(InputSplit split,
JobConf job,
Reporter reporter)
Get the
RecordReader for the given InputSplit . |
RecordReader<org.apache.hadoop.io.LongWritable,org.apache.hadoop.io.BytesWritable> |
FixedLengthInputFormat.getRecordReader(InputSplit genericSplit,
JobConf job,
Reporter reporter) |
abstract RecordReader<K,V> |
FileInputFormat.getRecordReader(InputSplit split,
JobConf job,
Reporter reporter) |
RecordWriter<K,V> |
TextOutputFormat.getRecordWriter(org.apache.hadoop.fs.FileSystem ignored,
JobConf job,
String name,
org.apache.hadoop.util.Progressable progress) |
RecordWriter<K,V> |
SequenceFileOutputFormat.getRecordWriter(org.apache.hadoop.fs.FileSystem ignored,
JobConf job,
String name,
org.apache.hadoop.util.Progressable progress) |
RecordWriter<org.apache.hadoop.io.BytesWritable,org.apache.hadoop.io.BytesWritable> |
SequenceFileAsBinaryOutputFormat.getRecordWriter(org.apache.hadoop.fs.FileSystem ignored,
JobConf job,
String name,
org.apache.hadoop.util.Progressable progress) |
RecordWriter<K,V> |
OutputFormat.getRecordWriter(org.apache.hadoop.fs.FileSystem ignored,
JobConf job,
String name,
org.apache.hadoop.util.Progressable progress)
Get the
RecordWriter for the given job. |
RecordWriter<org.apache.hadoop.io.WritableComparable,org.apache.hadoop.io.Writable> |
MapFileOutputFormat.getRecordWriter(org.apache.hadoop.fs.FileSystem ignored,
JobConf job,
String name,
org.apache.hadoop.util.Progressable progress) |
abstract RecordWriter<K,V> |
FileOutputFormat.getRecordWriter(org.apache.hadoop.fs.FileSystem ignored,
JobConf job,
String name,
org.apache.hadoop.util.Progressable progress) |
static Class<? extends org.apache.hadoop.io.WritableComparable> |
SequenceFileAsBinaryOutputFormat.getSequenceFileOutputKeyClass(JobConf conf)
Get the key class for the
SequenceFile |
static Class<? extends org.apache.hadoop.io.Writable> |
SequenceFileAsBinaryOutputFormat.getSequenceFileOutputValueClass(JobConf conf)
Get the value class for the
SequenceFile |
InputSplit[] |
MultiFileInputFormat.getSplits(JobConf job,
int numSplits) |
InputSplit[] |
InputFormat.getSplits(JobConf job,
int numSplits)
Logically split the set of input files for the job.
|
InputSplit[] |
FileInputFormat.getSplits(JobConf job,
int numSplits)
Splits files returned by
FileInputFormat.listStatus(JobConf) when
they're too big. |
static long |
TaskLog.getTaskLogLength(JobConf conf)
Get the desired maximum length of task's logs.
|
static JobClient.TaskStatusFilter |
JobClient.getTaskOutputFilter(JobConf job)
Get the task output filter out of the JobConf.
|
static org.apache.hadoop.fs.Path |
FileOutputFormat.getTaskOutputPath(JobConf conf,
String name)
Helper function to create the task's temporary output directory and
return the path to the task's output file.
|
static String |
FileOutputFormat.getUniqueName(JobConf conf,
String name)
Helper function to generate a name that is unique for the task.
|
static org.apache.hadoop.fs.Path |
FileOutputFormat.getWorkOutputPath(JobConf conf)
Get the
Path to the task's temporary output directory
for the map-reduce job
Tasks' Side-Effect Files |
void |
JobClient.init(JobConf conf)
Connect to the default cluster
|
void |
Task.initialize(JobConf job,
JobID id,
Reporter reporter,
boolean useNewApi) |
protected boolean |
Task.keepTaskFiles(JobConf conf) |
protected org.apache.hadoop.fs.FileStatus[] |
SequenceFileInputFormat.listStatus(JobConf job) |
protected org.apache.hadoop.fs.FileStatus[] |
FileInputFormat.listStatus(JobConf job)
List input directories.
|
void |
Task.localizeConfiguration(JobConf conf)
Localize the given JobConf to be specific for this task.
|
void |
ReduceTask.localizeConfiguration(JobConf conf)
Localize the given JobConf to be specific for this task.
|
void |
MapTask.localizeConfiguration(JobConf conf) |
static void |
JobEndNotifier.localRunnerNotification(JobConf conf,
JobStatus status) |
boolean |
JobClient.monitorAndPrintJob(JobConf conf,
RunningJob job)
Monitor a job and print status in real-time as progress is made and tasks
fail.
|
abstract void |
Task.run(JobConf job,
TaskUmbilicalProtocol umbilical)
Run this task as a part of the named job.
|
void |
ReduceTask.run(JobConf job,
TaskUmbilicalProtocol umbilical) |
void |
MapTask.run(JobConf job,
TaskUmbilicalProtocol umbilical) |
static RunningJob |
JobClient.runJob(JobConf job)
Utility that submits a job, then polls for progress until the job is
complete.
|
static void |
FileOutputFormat.setCompressOutput(JobConf conf,
boolean compress)
Set whether the output of the job is compressed.
|
static void |
FileInputFormat.setInputPathFilter(JobConf conf,
Class<? extends org.apache.hadoop.fs.PathFilter> filter)
Set a PathFilter to be applied to the input paths for the map-reduce job.
|
static void |
FileInputFormat.setInputPaths(JobConf conf,
org.apache.hadoop.fs.Path... inputPaths)
Set the array of
Path s as the list of inputs
for the map-reduce job. |
static void |
FileInputFormat.setInputPaths(JobConf conf,
String commaSeparatedPaths)
Sets the given comma separated paths as the list of inputs
for the map-reduce job.
|
static void |
SequenceFileOutputFormat.setOutputCompressionType(JobConf conf,
org.apache.hadoop.io.SequenceFile.CompressionType style)
Set the
SequenceFile.CompressionType for the output SequenceFile . |
static void |
FileOutputFormat.setOutputCompressorClass(JobConf conf,
Class<? extends org.apache.hadoop.io.compress.CompressionCodec> codecClass)
Set the
CompressionCodec to be used to compress job outputs. |
static void |
FileOutputFormat.setOutputPath(JobConf conf,
org.apache.hadoop.fs.Path outputDir)
Set the
Path of the output directory for the map-reduce job. |
static void |
SequenceFileAsBinaryOutputFormat.setSequenceFileOutputKeyClass(JobConf conf,
Class<?> theClass)
Set the key class for the
SequenceFile |
static void |
SequenceFileAsBinaryOutputFormat.setSequenceFileOutputValueClass(JobConf conf,
Class<?> theClass)
Set the value class for the
SequenceFile |
static void |
SkipBadRecords.setSkipOutputPath(JobConf conf,
org.apache.hadoop.fs.Path path)
Set the directory to which skipped records are written.
|
static void |
JobClient.setTaskOutputFilter(JobConf job,
JobClient.TaskStatusFilter newValue)
Modify the JobConf to set the task output filter.
|
static void |
FileOutputFormat.setWorkOutputPath(JobConf conf,
org.apache.hadoop.fs.Path outputDir)
Set the
Path of the task's temporary output directory
for the map-reduce job. |
RunningJob |
JobClient.submitJob(JobConf conf)
Submit a job to the MR system.
|
RunningJob |
JobClient.submitJobInternal(JobConf conf) |
void |
SpillRecord.writeToFile(org.apache.hadoop.fs.Path loc,
JobConf job)
Write this spill record to the location provided.
|
void |
SpillRecord.writeToFile(org.apache.hadoop.fs.Path loc,
JobConf job,
Checksum crc) |
Constructor and Description |
---|
FileSplit(org.apache.hadoop.fs.Path file,
long start,
long length,
JobConf conf)
Deprecated.
|
JobClient(JobConf conf)
Build a job client with the given
JobConf , and connect to the
default cluster |
JobContextImpl(JobConf conf,
JobID jobId) |
JobContextImpl(JobConf conf,
JobID jobId,
org.apache.hadoop.util.Progressable progress) |
MapOutputCollector.Context(MapTask mapTask,
JobConf jobConf,
Task.TaskReporter reporter) |
MultiFileSplit(JobConf job,
org.apache.hadoop.fs.Path[] files,
long[] lengths) |
ShuffleConsumerPlugin.Context(TaskAttemptID reduceId,
JobConf jobConf,
org.apache.hadoop.fs.FileSystem localFS,
TaskUmbilicalProtocol umbilical,
org.apache.hadoop.fs.LocalDirAllocator localDirAllocator,
Reporter reporter,
org.apache.hadoop.io.compress.CompressionCodec codec,
Class<? extends Reducer> combinerClass,
Task.CombineOutputCollector<K,V> combineCollector,
Counters.Counter spilledRecordsCounter,
Counters.Counter reduceCombineInputCounter,
Counters.Counter shuffledMapsCounter,
Counters.Counter reduceShuffleBytes,
Counters.Counter failedShuffleCounter,
Counters.Counter mergedMapOutputsCounter,
TaskStatus status,
org.apache.hadoop.util.Progress copyPhase,
org.apache.hadoop.util.Progress mergePhase,
Task reduceTask,
MapOutputFile mapOutputFile,
Map<TaskAttemptID,MapOutputFile> localMapFiles) |
SpillRecord(org.apache.hadoop.fs.Path indexFileName,
JobConf job) |
SpillRecord(org.apache.hadoop.fs.Path indexFileName,
JobConf job,
Checksum crc,
String expectedIndexOwner) |
SpillRecord(org.apache.hadoop.fs.Path indexFileName,
JobConf job,
String expectedIndexOwner) |
Task.OldCombinerRunner(Class<? extends Reducer<K,V,K,V>> cls,
JobConf conf,
Counters.Counter inputCounter,
Task.TaskReporter reporter) |
TaskAttemptContextImpl(JobConf conf,
TaskAttemptID taskid) |
Modifier and Type | Method and Description |
---|---|
JobConf |
Job.getJobConf() |
Modifier and Type | Method and Description |
---|---|
void |
Job.setJobConf(JobConf jobConf)
Set the mapred job conf for this job.
|
Constructor and Description |
---|
Job(JobConf conf) |
Job(JobConf jobConf,
ArrayList<?> dependingJobs)
Construct a job.
|
Modifier and Type | Method and Description |
---|---|
ComposableRecordReader<K,TupleWritable> |
CompositeInputFormat.getRecordReader(InputSplit split,
JobConf job,
Reporter reporter)
Construct a CompositeRecordReader for the children of this InputFormat
as defined in the init expression.
|
ComposableRecordReader<K,V> |
ComposableInputFormat.getRecordReader(InputSplit split,
JobConf job,
Reporter reporter) |
InputSplit[] |
CompositeInputFormat.getSplits(JobConf job,
int numSplits)
Build a CompositeInputSplit from the child InputFormats by assigning the
ith split from each child to the ith composite split.
|
void |
CompositeInputFormat.setFormat(JobConf job)
Interpret a given string as a composite expression.
|
Constructor and Description |
---|
JoinRecordReader(int id,
JobConf conf,
int capacity,
Class<? extends org.apache.hadoop.io.WritableComparator> cmpcl) |
MultiFilterRecordReader(int id,
JobConf conf,
int capacity,
Class<? extends org.apache.hadoop.io.WritableComparator> cmpcl) |
Modifier and Type | Field and Description |
---|---|
protected JobConf |
CombineFileRecordReader.jc |
Modifier and Type | Method and Description |
---|---|
JobConf |
CombineFileSplit.getJob() |
Modifier and Type | Method and Description |
---|---|
static void |
MultipleInputs.addInputPath(JobConf conf,
org.apache.hadoop.fs.Path path,
Class<? extends InputFormat> inputFormatClass)
Add a
Path with a custom InputFormat to the list of
inputs for the map-reduce job. |
static void |
MultipleInputs.addInputPath(JobConf conf,
org.apache.hadoop.fs.Path path,
Class<? extends InputFormat> inputFormatClass,
Class<? extends Mapper> mapperClass)
|
static <K1,V1,K2,V2> |
ChainReducer.addMapper(JobConf job,
Class<? extends Mapper<K1,V1,K2,V2>> klass,
Class<? extends K1> inputKeyClass,
Class<? extends V1> inputValueClass,
Class<? extends K2> outputKeyClass,
Class<? extends V2> outputValueClass,
boolean byValue,
JobConf mapperConf)
Adds a Mapper class to the chain job's JobConf.
|
static <K1,V1,K2,V2> |
ChainMapper.addMapper(JobConf job,
Class<? extends Mapper<K1,V1,K2,V2>> klass,
Class<? extends K1> inputKeyClass,
Class<? extends V1> inputValueClass,
Class<? extends K2> outputKeyClass,
Class<? extends V2> outputValueClass,
boolean byValue,
JobConf mapperConf)
Adds a Mapper class to the chain job's JobConf.
|
static void |
MultipleOutputs.addMultiNamedOutput(JobConf conf,
String namedOutput,
Class<? extends OutputFormat> outputFormatClass,
Class<?> keyClass,
Class<?> valueClass)
Adds a multi named output for the job.
|
static void |
MultipleOutputs.addNamedOutput(JobConf conf,
String namedOutput,
Class<? extends OutputFormat> outputFormatClass,
Class<?> keyClass,
Class<?> valueClass)
Adds a named output for the job.
|
void |
NullOutputFormat.checkOutputSpecs(org.apache.hadoop.fs.FileSystem ignored,
JobConf job) |
void |
LazyOutputFormat.checkOutputSpecs(org.apache.hadoop.fs.FileSystem ignored,
JobConf job) |
void |
FilterOutputFormat.checkOutputSpecs(org.apache.hadoop.fs.FileSystem ignored,
JobConf job) |
void |
TotalOrderPartitioner.configure(JobConf job) |
void |
MultithreadedMapRunner.configure(JobConf jobConf) |
void |
KeyFieldBasedComparator.configure(JobConf job) |
void |
FieldSelectionMapReduce.configure(JobConf job) |
void |
RegexMapper.configure(JobConf job) |
void |
NLineInputFormat.configure(JobConf conf) |
void |
KeyFieldBasedPartitioner.configure(JobConf job) |
void |
HashPartitioner.configure(JobConf job) |
void |
DelegatingMapper.configure(JobConf conf) |
void |
ChainReducer.configure(JobConf job)
Configures the ChainReducer, the Reducer and all the Mappers in the chain.
|
void |
ChainMapper.configure(JobConf job)
Configures the ChainMapper and all the Mappers in the chain.
|
void |
BinaryPartitioner.configure(JobConf job) |
protected void |
CombineFileInputFormat.createPool(JobConf conf,
List<org.apache.hadoop.fs.PathFilter> filters)
Deprecated.
|
protected void |
CombineFileInputFormat.createPool(JobConf conf,
org.apache.hadoop.fs.PathFilter... filters)
Deprecated.
|
protected abstract RecordWriter<K,V> |
MultipleOutputFormat.getBaseRecordWriter(org.apache.hadoop.fs.FileSystem fs,
JobConf job,
String name,
org.apache.hadoop.util.Progressable arg3) |
protected RecordWriter<K,V> |
MultipleTextOutputFormat.getBaseRecordWriter(org.apache.hadoop.fs.FileSystem fs,
JobConf job,
String name,
org.apache.hadoop.util.Progressable arg3) |
protected RecordWriter<K,V> |
MultipleSequenceFileOutputFormat.getBaseRecordWriter(org.apache.hadoop.fs.FileSystem fs,
JobConf job,
String name,
org.apache.hadoop.util.Progressable arg3) |
static boolean |
MultipleOutputs.getCountersEnabled(JobConf conf)
Returns if the counters for the named outputs are enabled or not.
|
protected String |
MultipleOutputFormat.getInputFileBasedOutputFileName(JobConf job,
String name)
Generate the outfile name based on a given anme and the input file name.
|
static Class<? extends OutputFormat> |
MultipleOutputs.getNamedOutputFormatClass(JobConf conf,
String namedOutput)
Returns the named output OutputFormat.
|
static Class<?> |
MultipleOutputs.getNamedOutputKeyClass(JobConf conf,
String namedOutput)
Returns the key class for a named output.
|
static List<String> |
MultipleOutputs.getNamedOutputsList(JobConf conf)
Returns list of channel names.
|
static Class<?> |
MultipleOutputs.getNamedOutputValueClass(JobConf conf,
String namedOutput)
Returns the value class for a named output.
|
static String |
TotalOrderPartitioner.getPartitionFile(JobConf job)
Deprecated.
|
RecordReader<org.apache.hadoop.io.LongWritable,org.apache.hadoop.io.Text> |
NLineInputFormat.getRecordReader(InputSplit genericSplit,
JobConf job,
Reporter reporter) |
RecordReader<K,V> |
DelegatingInputFormat.getRecordReader(InputSplit split,
JobConf conf,
Reporter reporter) |
RecordReader<org.apache.hadoop.io.LongWritable,org.apache.hadoop.io.Text> |
CombineTextInputFormat.getRecordReader(InputSplit split,
JobConf conf,
Reporter reporter) |
RecordReader<K,V> |
CombineSequenceFileInputFormat.getRecordReader(InputSplit split,
JobConf conf,
Reporter reporter) |
abstract RecordReader<K,V> |
CombineFileInputFormat.getRecordReader(InputSplit split,
JobConf job,
Reporter reporter)
This is not implemented yet.
|
RecordWriter<K,V> |
MultipleOutputFormat.getRecordWriter(org.apache.hadoop.fs.FileSystem fs,
JobConf job,
String name,
org.apache.hadoop.util.Progressable arg3)
Create a composite record writer that can write key/value data to different
output files
|
RecordWriter<K,V> |
NullOutputFormat.getRecordWriter(org.apache.hadoop.fs.FileSystem ignored,
JobConf job,
String name,
org.apache.hadoop.util.Progressable progress) |
RecordWriter<K,V> |
LazyOutputFormat.getRecordWriter(org.apache.hadoop.fs.FileSystem ignored,
JobConf job,
String name,
org.apache.hadoop.util.Progressable progress) |
RecordWriter<K,V> |
FilterOutputFormat.getRecordWriter(org.apache.hadoop.fs.FileSystem ignored,
JobConf job,
String name,
org.apache.hadoop.util.Progressable progress) |
K[] |
InputSampler.Sampler.getSample(InputFormat<K,V> inf,
JobConf job)
For a given job, collect and return a subset of the keys from the
input data.
|
K[] |
InputSampler.SplitSampler.getSample(InputFormat<K,V> inf,
JobConf job)
From each split sampled, take the first numSamples / numSplits records.
|
K[] |
InputSampler.RandomSampler.getSample(InputFormat<K,V> inf,
JobConf job)
Randomize the split order, then take the specified number of keys from
each split sampled, where each key is selected with the specified
probability and possibly replaced by a subsequently selected key when
the quota of keys from that split is satisfied.
|
K[] |
InputSampler.IntervalSampler.getSample(InputFormat<K,V> inf,
JobConf job)
For each split sampled, emit when the ratio of the number of records
retained to the total record count is less than the specified
frequency.
|
InputSplit[] |
NLineInputFormat.getSplits(JobConf job,
int numSplits)
Logically splits the set of input files for the job, splits N lines
of the input as one split.
|
InputSplit[] |
DelegatingInputFormat.getSplits(JobConf conf,
int numSplits) |
InputSplit[] |
CombineFileInputFormat.getSplits(JobConf job,
int numSplits) |
static boolean |
MultipleOutputs.isMultiNamedOutput(JobConf conf,
String namedOutput)
Returns if a named output is multiple.
|
protected org.apache.hadoop.fs.FileStatus[] |
CombineFileInputFormat.listStatus(JobConf job)
List input directories.
|
static void |
MultipleOutputs.setCountersEnabled(JobConf conf,
boolean enabled)
Enables or disables counters for the named outputs.
|
static void |
LazyOutputFormat.setOutputFormatClass(JobConf job,
Class<? extends OutputFormat> theClass)
Set the underlying output format for LazyOutputFormat.
|
static void |
TotalOrderPartitioner.setPartitionFile(JobConf job,
org.apache.hadoop.fs.Path p)
Deprecated.
|
static <K1,V1,K2,V2> |
ChainReducer.setReducer(JobConf job,
Class<? extends Reducer<K1,V1,K2,V2>> klass,
Class<? extends K1> inputKeyClass,
Class<? extends V1> inputValueClass,
Class<? extends K2> outputKeyClass,
Class<? extends V2> outputValueClass,
boolean byValue,
JobConf reducerConf)
Sets the Reducer class to the chain job's JobConf.
|
static <K,V> void |
InputSampler.writePartitionFile(JobConf job,
InputSampler.Sampler<K,V> sampler) |
Constructor and Description |
---|
CombineFileRecordReader(JobConf job,
CombineFileSplit split,
Reporter reporter,
Class<RecordReader<K,V>> rrClass)
A generic RecordReader that can hand out different recordReaders
for each chunk in the CombineFileSplit.
|
CombineFileSplit(JobConf job,
org.apache.hadoop.fs.Path[] files,
long[] lengths) |
CombineFileSplit(JobConf job,
org.apache.hadoop.fs.Path[] files,
long[] start,
long[] lengths,
String[] locations) |
InputSampler(JobConf conf) |
MultipleOutputs(JobConf job)
Creates and initializes multiple named outputs support, it should be
instantiated in the Mapper/Reducer configure method.
|
Modifier and Type | Method and Description |
---|---|
static JobConf |
ValueAggregatorJob.createValueAggregatorJob(String[] args)
Create an Aggregate based map/reduce job.
|
static JobConf |
ValueAggregatorJob.createValueAggregatorJob(String[] args,
Class<?> caller)
Create an Aggregate based map/reduce job.
|
static JobConf |
ValueAggregatorJob.createValueAggregatorJob(String[] args,
Class<? extends ValueAggregatorDescriptor>[] descriptors) |
static JobConf |
ValueAggregatorJob.createValueAggregatorJob(String[] args,
Class<? extends ValueAggregatorDescriptor>[] descriptors,
Class<?> caller) |
Modifier and Type | Method and Description |
---|---|
void |
ValueAggregatorJobBase.configure(JobConf job) |
void |
ValueAggregatorDescriptor.configure(JobConf job)
Configure the object
|
void |
ValueAggregatorCombiner.configure(JobConf job)
Combiner does not need to configure.
|
void |
ValueAggregatorBaseDescriptor.configure(JobConf job)
get the input file name.
|
void |
UserDefinedValueAggregatorDescriptor.configure(JobConf job)
Do nothing.
|
static void |
ValueAggregatorJob.setAggregatorDescriptors(JobConf job,
Class<? extends ValueAggregatorDescriptor>[] descriptors) |
Constructor and Description |
---|
UserDefinedValueAggregatorDescriptor(String className,
JobConf job) |
Modifier and Type | Method and Description |
---|---|
void |
DBOutputFormat.checkOutputSpecs(org.apache.hadoop.fs.FileSystem filesystem,
JobConf job)
Check for validity of the output-specification for the job.
|
void |
DBInputFormat.configure(JobConf job)
Initializes a new instance from a
JobConf . |
static void |
DBConfiguration.configureDB(JobConf job,
String driverClass,
String dbUrl)
Sets the DB access related fields in the JobConf.
|
static void |
DBConfiguration.configureDB(JobConf job,
String driverClass,
String dbUrl,
String userName,
String passwd)
Sets the DB access related fields in the JobConf.
|
RecordReader<org.apache.hadoop.io.LongWritable,T> |
DBInputFormat.getRecordReader(InputSplit split,
JobConf job,
Reporter reporter)
Get the
RecordReader for the given InputSplit . |
RecordWriter<K,V> |
DBOutputFormat.getRecordWriter(org.apache.hadoop.fs.FileSystem filesystem,
JobConf job,
String name,
org.apache.hadoop.util.Progressable progress)
Get the
RecordWriter for the given job. |
InputSplit[] |
DBInputFormat.getSplits(JobConf job,
int chunks)
Logically split the set of input files for the job.
|
static void |
DBInputFormat.setInput(JobConf job,
Class<? extends DBWritable> inputClass,
String inputQuery,
String inputCountQuery)
Initializes the map-part of the job with the appropriate input settings.
|
static void |
DBInputFormat.setInput(JobConf job,
Class<? extends DBWritable> inputClass,
String tableName,
String conditions,
String orderBy,
String... fieldNames)
Initializes the map-part of the job with the appropriate input settings.
|
static void |
DBOutputFormat.setOutput(JobConf job,
String tableName,
int fieldCount)
Initializes the reduce-part of the job with the appropriate output settings
|
static void |
DBOutputFormat.setOutput(JobConf job,
String tableName,
String... fieldNames)
Initializes the reduce-part of the job with the appropriate output settings
|
Constructor and Description |
---|
DBInputFormat.DBRecordReader(DBInputFormat.DBInputSplit split,
Class<T> inputClass,
JobConf job)
The constructor is kept to be compatible with M/R 1.x
|
DBInputFormat.DBRecordReader(DBInputFormat.DBInputSplit split,
Class<T> inputClass,
JobConf job,
Connection conn,
DBConfiguration dbConfig,
String cond,
String[] fields,
String table) |
Modifier and Type | Method and Description |
---|---|
static String |
Submitter.getExecutable(JobConf conf)
Get the URI of the application's executable.
|
static boolean |
Submitter.getIsJavaMapper(JobConf conf)
Check whether the job is using a Java Mapper.
|
static boolean |
Submitter.getIsJavaRecordReader(JobConf conf)
Check whether the job is using a Java RecordReader
|
static boolean |
Submitter.getIsJavaRecordWriter(JobConf conf)
Will the reduce use a Java RecordWriter?
|
static boolean |
Submitter.getIsJavaReducer(JobConf conf)
Check whether the job is using a Java Reducer.
|
static boolean |
Submitter.getKeepCommandFile(JobConf conf)
Does the user want to keep the command file for debugging? If this is
true, pipes will write a copy of the command data to a file in the
task directory named "downlink.data", which may be used to run the C++
program under the debugger.
|
static RunningJob |
Submitter.jobSubmit(JobConf conf)
Submit a job to the Map-Reduce framework.
|
static RunningJob |
Submitter.runJob(JobConf conf)
Submit a job to the map/reduce cluster.
|
static void |
Submitter.setExecutable(JobConf conf,
String executable)
Set the URI for the application's executable.
|
static void |
Submitter.setIsJavaMapper(JobConf conf,
boolean value)
Set whether the Mapper is written in Java.
|
static void |
Submitter.setIsJavaRecordReader(JobConf conf,
boolean value)
Set whether the job is using a Java RecordReader.
|
static void |
Submitter.setIsJavaRecordWriter(JobConf conf,
boolean value)
Set whether the job will use a Java RecordWriter.
|
static void |
Submitter.setIsJavaReducer(JobConf conf,
boolean value)
Set whether the Reducer is written in Java.
|
static void |
Submitter.setKeepCommandFile(JobConf conf,
boolean keep)
Set whether to keep the command file for debugging
|
static RunningJob |
Submitter.submitJob(JobConf conf)
Deprecated.
|
Modifier and Type | Method and Description |
---|---|
JobConf |
JobSubmittedEvent.getJobConf() |
Constructor and Description |
---|
JobSubmittedEvent(JobID id,
String jobName,
String userName,
long submitTime,
String jobConfPath,
Map<JobACL,org.apache.hadoop.security.authorize.AccessControlList> jobACLs,
String jobQueueName,
String workflowId,
String workflowName,
String workflowNodeName,
String workflowAdjacencies,
String workflowTags,
JobConf conf)
Create an event to record job submission
|
Modifier and Type | Method and Description |
---|---|
static org.apache.hadoop.security.Credentials |
TokenCache.loadTokens(String jobTokenFile,
JobConf conf)
Deprecated.
Use
Credentials.readTokenStorageFile(org.apache.hadoop.fs.Path, org.apache.hadoop.conf.Configuration) instead,
this method is included for compatibility against Hadoop-1. |
Modifier and Type | Field and Description |
---|---|
protected JobConf |
JobContextImpl.conf |
Constructor and Description |
---|
MergeManagerImpl(TaskAttemptID reduceId,
JobConf jobConf,
org.apache.hadoop.fs.FileSystem localFS,
org.apache.hadoop.fs.LocalDirAllocator localDirAllocator,
Reporter reporter,
org.apache.hadoop.io.compress.CompressionCodec codec,
Class<? extends Reducer> combinerClass,
Task.CombineOutputCollector<K,V> combineCollector,
Counters.Counter spilledRecordsCounter,
Counters.Counter reduceCombineInputCounter,
Counters.Counter mergedMapOutputsCounter,
ExceptionReporter exceptionReporter,
org.apache.hadoop.util.Progress mergePhase,
MapOutputFile mapOutputFile) |
ShuffleSchedulerImpl(JobConf job,
TaskStatus status,
TaskAttemptID reduceId,
ExceptionReporter reporter,
org.apache.hadoop.util.Progress progress,
Counters.Counter shuffledMapsCounter,
Counters.Counter reduceShuffleBytes,
Counters.Counter failedShuffleCounter) |
Copyright © 2020 Apache Software Foundation. All rights reserved.