Class QueryChangeStreamAction
- java.lang.Object
-
- org.apache.beam.sdk.io.gcp.spanner.changestreams.action.QueryChangeStreamAction
-
public class QueryChangeStreamAction extends java.lang.Object
Main action class for querying a partition change stream. This class will perform the change stream query and depending on the record type received, it will dispatch the processing of it to one of the following:ChildPartitionsRecordAction
,HeartbeatRecordAction
orDataChangeRecordAction
.This class will also make sure to mirror the current watermark (event timestamp processed) in the Connector's metadata tables, by registering a bundle after commit action.
When the change stream query for the partition is finished, this class will update the state of the partition in the metadata tables as FINISHED, indicating completion.
-
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description org.apache.beam.sdk.transforms.DoFn.ProcessContinuation
run(PartitionMetadata partition, org.apache.beam.sdk.transforms.splittabledofn.RestrictionTracker<TimestampRange,com.google.cloud.Timestamp> tracker, org.apache.beam.sdk.transforms.DoFn.OutputReceiver<DataChangeRecord> receiver, org.apache.beam.sdk.transforms.splittabledofn.ManualWatermarkEstimator<org.joda.time.Instant> watermarkEstimator, org.apache.beam.sdk.transforms.DoFn.BundleFinalizer bundleFinalizer)
This method will dispatch a change stream query for the given partition, it delegate the processing of the records to one of the corresponding action classes registered and it will keep the state of the partition up to date in the Connector's metadata table.
-
-
-
Method Detail
-
run
public org.apache.beam.sdk.transforms.DoFn.ProcessContinuation run(PartitionMetadata partition, org.apache.beam.sdk.transforms.splittabledofn.RestrictionTracker<TimestampRange,com.google.cloud.Timestamp> tracker, org.apache.beam.sdk.transforms.DoFn.OutputReceiver<DataChangeRecord> receiver, org.apache.beam.sdk.transforms.splittabledofn.ManualWatermarkEstimator<org.joda.time.Instant> watermarkEstimator, org.apache.beam.sdk.transforms.DoFn.BundleFinalizer bundleFinalizer)
This method will dispatch a change stream query for the given partition, it delegate the processing of the records to one of the corresponding action classes registered and it will keep the state of the partition up to date in the Connector's metadata table.The algorithm is as follows:
- A change stream query for the partition is performed.
- For each record, we check the type of the record and dispatch the processing to one of the actions registered.
- If an
Optional
with aDoFn.ProcessContinuation.stop()
is returned from the actions, we stop processing and return. - Before returning we register a bundle finalizer callback to update the watermark of the partition in the metadata tables to the latest processed timestamp.
- When a change stream query finishes successfully (no more records) we update the partition state to FINISHED.
- Parameters:
partition
- the current partition being processedtracker
- the restriction tracker of theReadChangeStreamPartitionDoFn
SDFreceiver
- the output receiver of theReadChangeStreamPartitionDoFn
SDFwatermarkEstimator
- the watermark estimator of theReadChangeStreamPartitionDoFn
SDFbundleFinalizer
- the bundle finalizer forReadChangeStreamPartitionDoFn
SDF bundles- Returns:
- a
DoFn.ProcessContinuation.stop()
if a record timestamp could not be claimed or if the partition processing has finished
-
-