org.apache.hadoop.mapred.join

Class CompositeInputFormat<K extends org.apache.hadoop.io.WritableComparable>

java.lang.Object

org.apache.hadoop.mapred.join.CompositeInputFormat<K>

All Implemented Interfaces:

InputFormat<K,TupleWritable>, ComposableInputFormat<K,TupleWritable>

@InterfaceAudience.Public
 @InterfaceStability.Stable
public class CompositeInputFormat<K extends org.apache.hadoop.io.WritableComparable>
extends Object
implements ComposableInputFormat<K,TupleWritable>

An InputFormat capable of performing joins over a set of data sources sorted and partitioned the same way. A user may define new join types by setting the property mapred.join.define.<ident> to a classname. In the expression mapred.join.expr, the identifier will be assumed to be a ComposableRecordReader. mapred.join.keycomparator can be a classname used to compare keys in the join.

See Also:

setFormat(org.apache.hadoop.mapred.JobConf), JoinRecordReader, MultiFilterRecordReader

Constructor Summary

Constructors

Constructor and Description
CompositeInputFormat()

Method Summary

Modifier and Type	Method and Description
protected void	addDefaults() Adds the default set of identifiers to the parser.
static String	compose(Class<? extends InputFormat> inf, String path) Convenience method for constructing composite formats.
static String	compose(String op, Class<? extends InputFormat> inf, org.apache.hadoop.fs.Path... path) Convenience method for constructing composite formats.
static String	compose(String op, Class<? extends InputFormat> inf, String... path) Convenience method for constructing composite formats.
ComposableRecordReader<K,TupleWritable>	getRecordReader(InputSplit split, JobConf job, Reporter reporter) Construct a CompositeRecordReader for the children of this InputFormat as defined in the init expression.
InputSplit[]	getSplits(JobConf job, int numSplits) Build a CompositeInputSplit from the child InputFormats by assigning the ith split from each child to the ith composite split.
void	setFormat(JobConf job) Interpret a given string as a composite expression.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

CompositeInputFormat

public CompositeInputFormat()

Method Detail

setFormat

public void setFormat(JobConf job)
               throws IOException

Interpret a given string as a composite expression. func ::= <ident>([<func>,]*<func>) func ::= tbl(<class>,"<path>") class ::= @see java.lang.Class#forName(java.lang.String) path ::= @see org.apache.hadoop.fs.Path#Path(java.lang.String) Reads expression from the mapred.join.expr property and user-supplied join types from mapred.join.define.<ident> types. Paths supplied to tbl are given as input paths to the InputFormat class listed.

Throws:

IOException

See Also:

compose(java.lang.String, java.lang.Class, java.lang.String...)

addDefaults

protected void addDefaults()

Adds the default set of identifiers to the parser.

getSplits

public InputSplit[] getSplits(JobConf job,
                              int numSplits)
                       throws IOException

Build a CompositeInputSplit from the child InputFormats by assigning the ith split from each child to the ith composite split.

Specified by:

getSplits in interface InputFormat<K extends org.apache.hadoop.io.WritableComparable,TupleWritable>

Parameters:

job - job configuration.

numSplits - the desired number of splits, a hint.

Returns:

an array of InputSplits for the job.

Throws:

IOException

getRecordReader

public ComposableRecordReader<K,TupleWritable> getRecordReader(InputSplit split,
                                                               JobConf job,
                                                               Reporter reporter)
                                                        throws IOException

Construct a CompositeRecordReader for the children of this InputFormat as defined in the init expression. The outermost join need only be composable, not necessarily a composite. Mandating TupleWritable isn't strictly correct.

Specified by:

getRecordReader in interface InputFormat<K extends org.apache.hadoop.io.WritableComparable,TupleWritable>

Specified by:

getRecordReader in interface ComposableInputFormat<K extends org.apache.hadoop.io.WritableComparable,TupleWritable>

Parameters:

split - the InputSplit

job - the job that this split belongs to

Returns:

a RecordReader

Throws:

IOException

compose

public static String compose(Class<? extends InputFormat> inf,
                             String path)

Convenience method for constructing composite formats. Given InputFormat class (inf), path (p) return: tbl(<inf>, <p>)

compose

public static String compose(String op,
                             Class<? extends InputFormat> inf,
                             String... path)

Convenience method for constructing composite formats. Given operation (op), Object class (inf), set of paths (p) return: <op>(tbl(<inf>,<p1>),tbl(<inf>,<p2>),...,tbl(<inf>,<pn>))

compose

public static String compose(String op,
                             Class<? extends InputFormat> inf,
                             org.apache.hadoop.fs.Path... path)

Convenience method for constructing composite formats. Given operation (op), Object class (inf), set of paths (p) return: <op>(tbl(<inf>,<p1>),tbl(<inf>,<p2>),...,tbl(<inf>,<pn>))