org.apache.hudi.utilities.deltastreamer

Class HoodieDeltaStreamer

java.lang.Object

org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer

All Implemented Interfaces:

Serializable

public class HoodieDeltaStreamer
extends Object
implements Serializable

An Utility which can incrementally take the output from HiveIncrementalPuller and apply it to the target table. Does not maintain any state, queries at runtime to see how far behind the target table is from the source table. This can be overriden to force sync from a timestamp.

In continuous mode, DeltaStreamer runs in loop-mode going through the below operations (a) pull-from-source (b) write-to-sink (c) Schedule Compactions if needed (d) Conditionally Sync to Hive each cycle. For MOR table with continuous mode enabled, a separate compactor thread is allocated to execute compactions

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	HoodieDeltaStreamer.Config
static class	HoodieDeltaStreamer.DeltaSyncService Syncs data either in single-run or in continuous mode.

Field Summary

Fields

Modifier and Type	Field and Description
protected HoodieDeltaStreamer.Config	cfg
static String	CHECKPOINT_KEY
static String	CHECKPOINT_RESET_KEY
static String	DELTASYNC_POOL_NAME
protected Option<HoodieDeltaStreamer.DeltaSyncService>	deltaSyncService

Constructor Summary

Constructors

Constructor and Description
HoodieDeltaStreamer(HoodieDeltaStreamer.Config cfg, org.apache.spark.api.java.JavaSparkContext jssc)
HoodieDeltaStreamer(HoodieDeltaStreamer.Config cfg, org.apache.spark.api.java.JavaSparkContext jssc, org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.conf.Configuration conf)
HoodieDeltaStreamer(HoodieDeltaStreamer.Config cfg, org.apache.spark.api.java.JavaSparkContext jssc, org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.conf.Configuration conf, Option<TypedProperties> props)
HoodieDeltaStreamer(HoodieDeltaStreamer.Config cfg, org.apache.spark.api.java.JavaSparkContext jssc, Option<TypedProperties> props)

Method Summary

Modifier and Type	Method and Description
HoodieDeltaStreamer.Config	getConfig()
static HoodieDeltaStreamer.Config	getConfig(String[] args)
HoodieDeltaStreamer.DeltaSyncService	getDeltaSyncService()
static void	main(String[] args)
void	shutdownGracefully()
void	sync() Main method to start syncing.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

CHECKPOINT_KEY

public static final String CHECKPOINT_KEY

See Also:

Constant Field Values

CHECKPOINT_RESET_KEY

public static final String CHECKPOINT_RESET_KEY

See Also:

Constant Field Values

cfg

protected final transient HoodieDeltaStreamer.Config cfg

deltaSyncService

protected transient Option<HoodieDeltaStreamer.DeltaSyncService> deltaSyncService

DELTASYNC_POOL_NAME

public static final String DELTASYNC_POOL_NAME

See Also:

Constant Field Values

Constructor Detail

HoodieDeltaStreamer

public HoodieDeltaStreamer(HoodieDeltaStreamer.Config cfg,
                           org.apache.spark.api.java.JavaSparkContext jssc)
                    throws IOException

Throws:

IOException

HoodieDeltaStreamer

public HoodieDeltaStreamer(HoodieDeltaStreamer.Config cfg,
                           org.apache.spark.api.java.JavaSparkContext jssc,
                           Option<TypedProperties> props)
                    throws IOException

Throws:

IOException

HoodieDeltaStreamer

public HoodieDeltaStreamer(HoodieDeltaStreamer.Config cfg,
                           org.apache.spark.api.java.JavaSparkContext jssc,
                           org.apache.hadoop.fs.FileSystem fs,
                           org.apache.hadoop.conf.Configuration conf)
                    throws IOException

Throws:

IOException

HoodieDeltaStreamer

public HoodieDeltaStreamer(HoodieDeltaStreamer.Config cfg,
                           org.apache.spark.api.java.JavaSparkContext jssc,
                           org.apache.hadoop.fs.FileSystem fs,
                           org.apache.hadoop.conf.Configuration conf,
                           Option<TypedProperties> props)
                    throws IOException

Throws:

IOException

Method Detail

shutdownGracefully

public void shutdownGracefully()

sync

public void sync()
          throws Exception

Main method to start syncing.

Throws:

Exception

getConfig

public HoodieDeltaStreamer.Config getConfig()

getConfig

public static final HoodieDeltaStreamer.Config getConfig(String[] args)

main

public static void main(String[] args)
                 throws Exception

Throws:

Exception

getDeltaSyncService

public HoodieDeltaStreamer.DeltaSyncService getDeltaSyncService()