@Internal
public class DetectNewPartitionsAction
extends java.lang.Object
DetectNewPartitionsDoFn
.Constructor and Description |
---|
DetectNewPartitionsAction(ChangeStreamMetrics metrics,
MetadataTableDao metadataTableDao,
org.joda.time.Instant endTime,
GenerateInitialPartitionsAction generateInitialPartitionsAction,
ResumeFromPreviousPipelineAction resumeFromPreviousPipelineAction,
ProcessNewPartitionsAction processNewPartitionsAction) |
Modifier and Type | Method and Description |
---|---|
org.apache.beam.sdk.transforms.DoFn.ProcessContinuation |
run(org.apache.beam.sdk.transforms.splittabledofn.RestrictionTracker<org.apache.beam.sdk.io.range.OffsetRange,java.lang.Long> tracker,
org.apache.beam.sdk.transforms.DoFn.OutputReceiver<PartitionRecord> receiver,
org.apache.beam.sdk.transforms.splittabledofn.ManualWatermarkEstimator<org.joda.time.Instant> watermarkEstimator,
InitialPipelineState initialPipelineState)
Perform the necessary steps to manage initial set of partitions and new partitions.
|
public DetectNewPartitionsAction(ChangeStreamMetrics metrics, MetadataTableDao metadataTableDao, @Nullable org.joda.time.Instant endTime, GenerateInitialPartitionsAction generateInitialPartitionsAction, ResumeFromPreviousPipelineAction resumeFromPreviousPipelineAction, ProcessNewPartitionsAction processNewPartitionsAction)
public org.apache.beam.sdk.transforms.DoFn.ProcessContinuation run(org.apache.beam.sdk.transforms.splittabledofn.RestrictionTracker<org.apache.beam.sdk.io.range.OffsetRange,java.lang.Long> tracker, org.apache.beam.sdk.transforms.DoFn.OutputReceiver<PartitionRecord> receiver, org.apache.beam.sdk.transforms.splittabledofn.ManualWatermarkEstimator<org.joda.time.Instant> watermarkEstimator, InitialPipelineState initialPipelineState) throws java.lang.Exception
tracker
- offset tracker that simply increment by 1 every single runreceiver
- output new partitionswatermarkEstimator
- update watermark that is a representation of the low watermark of the
entire beam pipelineDoFn.ProcessContinuation.resume()
with 1-second delay if the stream continues,
otherwise DoFn.ProcessContinuation.stop()
com.google.protobuf.InvalidProtocolBufferException
- if failing to process new partitionsjava.lang.Exception