org.apache.hudi.utilities.deltastreamer

Class HoodieDeltaStreamer.DeltaSyncService

java.lang.Object

org.apache.hudi.async.HoodieAsyncService

org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.DeltaSyncService

All Implemented Interfaces:

Serializable

Enclosing class:

HoodieDeltaStreamer

public static class HoodieDeltaStreamer.DeltaSyncService
extends HoodieAsyncService

Syncs data either in single-run or in continuous mode.

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
DeltaSyncService(HoodieDeltaStreamer.Config cfg, org.apache.spark.api.java.JavaSparkContext jssc, org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.conf.Configuration conf)
DeltaSyncService(HoodieDeltaStreamer.Config cfg, org.apache.spark.api.java.JavaSparkContext jssc, org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.conf.Configuration conf, Option<TypedProperties> properties)

Method Summary

Modifier and Type	Method and Description
void	close() Close all resources.
DeltaSync	getDeltaSync()
TypedProperties	getProps()
SchemaProvider	getSchemaProvider()
org.apache.spark.sql.SparkSession	getSparkSession()
protected Boolean	onInitializingWriteClient(SparkRDDWriteClient writeClient) Callback to initialize write client and start compaction service if required.
protected Pair<CompletableFuture,ExecutorService>	startService()

Methods inherited from class org.apache.hudi.async.HoodieAsyncService

enqueuePendingAsyncServiceInstant, isRunInDaemonMode, isShutdown, isShutdownRequested, shutdown, start, waitForShutdown, waitTillPendingAsyncServiceInstantsReducesTo

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DeltaSyncService

public DeltaSyncService(HoodieDeltaStreamer.Config cfg,
                        org.apache.spark.api.java.JavaSparkContext jssc,
                        org.apache.hadoop.fs.FileSystem fs,
                        org.apache.hadoop.conf.Configuration conf,
                        Option<TypedProperties> properties)
                 throws IOException

Throws:

IOException

DeltaSyncService

public DeltaSyncService(HoodieDeltaStreamer.Config cfg,
                        org.apache.spark.api.java.JavaSparkContext jssc,
                        org.apache.hadoop.fs.FileSystem fs,
                        org.apache.hadoop.conf.Configuration conf)
                 throws IOException

Throws:

IOException

Method Detail

getDeltaSync

public DeltaSync getDeltaSync()

startService

protected Pair<CompletableFuture,ExecutorService> startService()

Specified by:

startService in class HoodieAsyncService

onInitializingWriteClient

protected Boolean onInitializingWriteClient(SparkRDDWriteClient writeClient)

Callback to initialize write client and start compaction service if required.

Parameters:

writeClient - HoodieWriteClient

Returns:

close

public void close()

Close all resources.

getSchemaProvider

public SchemaProvider getSchemaProvider()

getSparkSession

public org.apache.spark.sql.SparkSession getSparkSession()

getProps

public TypedProperties getProps()