io.cdap.cdap.etl.api.batch

Class BatchSource<KEY_IN,VAL_IN,OUT>

java.lang.Object

io.cdap.cdap.etl.api.batch.BatchConfigurable<BatchSourceContext>

io.cdap.cdap.etl.api.batch.BatchSource<KEY_IN,VAL_IN,OUT>

Type Parameters:

KEY_IN - the type of input key from the Batch run

VAL_IN - the type of input value from the Batch run

OUT - the type of output for the source

All Implemented Interfaces:

Destroyable, PipelineConfigurable, StageLifecycle<BatchRuntimeContext>, SubmitterLifecycle<BatchSourceContext>, Transformation<KeyValue<KEY_IN,VAL_IN>,OUT>

@Beta
public abstract class BatchSource<KEY_IN,VAL_IN,OUT>
extends BatchConfigurable<BatchSourceContext>
implements Transformation<KeyValue<KEY_IN,VAL_IN>,OUT>, StageLifecycle<BatchRuntimeContext>

Batch Source forms the first stage of a Batch ETL Pipeline. In addition to configuring the Batch run, it also transforms the key value pairs provided by the Batch run into a single output type to be consumed by subsequent transforms. By default, the value of the key value pair will be emitted. initialize(io.cdap.cdap.etl.api.batch.BatchRuntimeContext), transform(io.cdap.cdap.api.dataset.lib.KeyValue<KEY_IN, VAL_IN>, io.cdap.cdap.etl.api.Emitter<OUT>) and destroy() methods are called inside the Batch Run while BatchConfigurable.prepareRun(T) and BatchConfigurable.onRunFinish(boolean, T) methods are called on the client side, which launches the Batch run, before the Batch run starts and after it finishes respectively.

Field Summary

Fields

Modifier and Type	Field and Description
static String	FORMAT_PLUGIN_TYPE
static String	PLUGIN_TYPE

Constructor Summary

Constructors

Constructor and Description
BatchSource()

Method Summary

Modifier and Type	Method and Description
void	destroy() Destroy the Batch Source stage.
void	initialize(BatchRuntimeContext context) Initialize the Batch Source stage.
void	transform(KeyValue<KEY_IN,VAL_IN> input, Emitter<OUT> emitter) Transform the KeyValue pair produced by the input, as set in BatchConfigurable.prepareRun(T), to a single object and emit it to the next stage.

Methods inherited from class io.cdap.cdap.etl.api.batch.BatchConfigurable

configurePipeline, onRunFinish, prepareRun

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

PLUGIN_TYPE

public static final String PLUGIN_TYPE

See Also:

Constant Field Values

FORMAT_PLUGIN_TYPE

public static final String FORMAT_PLUGIN_TYPE

See Also:

Constant Field Values

Constructor Detail

BatchSource

public BatchSource()

Method Detail

initialize

public void initialize(BatchRuntimeContext context)
                throws Exception

Initialize the Batch Source stage. Executed inside the Batch Run. This method is guaranteed to be invoked before any calls to transform(io.cdap.cdap.api.dataset.lib.KeyValue<KEY_IN, VAL_IN>, io.cdap.cdap.etl.api.Emitter<OUT>) are made.

Specified by:

initialize in interface StageLifecycle<BatchRuntimeContext>

Parameters:

context - BatchRuntimeContext

Throws:

Exception - if there is any error during initialization

transform

public void transform(KeyValue<KEY_IN,VAL_IN> input,
                      Emitter<OUT> emitter)
               throws Exception

Transform the KeyValue pair produced by the input, as set in BatchConfigurable.prepareRun(T), to a single object and emit it to the next stage. By default it emits the value. This method is invoked inside the Batch run.

Specified by:

transform in interface Transformation<KeyValue<KEY_IN,VAL_IN>,OUT>

Parameters:

input - the input to transform

emitter - Emitter to emit data to the next stage

Throws:

Exception - if there's an error during this method invocation

destroy

public void destroy()

Destroy the Batch Source stage. Executed at the end of the Batch run.

Specified by:

destroy in interface Destroyable