Package org.apache.beam.sdk.io.gcp.spanner.changestreams.dofn

Class PostProcessingMetricsDoFn

java.lang.Object

org.apache.beam.sdk.transforms.DoFn<DataChangeRecord,DataChangeRecord>

org.apache.beam.sdk.io.gcp.spanner.changestreams.dofn.PostProcessingMetricsDoFn

All Implemented Interfaces:

java.io.Serializable, org.apache.beam.sdk.transforms.display.HasDisplayData

public class PostProcessingMetricsDoFn
extends org.apache.beam.sdk.transforms.DoFn<DataChangeRecord,DataChangeRecord>
implements java.io.Serializable

A DoFn class to gather metrics about the emitted DataChangeRecords. It will simply delegate the metrics gathering to the ChangeStreamMetrics. The metrics measured in this component are:

1. The number of data records emitted.
2. The latency between a record's Cloud Spanner commit timestamp and the time it reached this component (referred as emit timestamp).
3. The streaming latency of a record from the change stream query.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.beam.sdk.transforms.DoFn

org.apache.beam.sdk.transforms.DoFn.AlwaysFetched, org.apache.beam.sdk.transforms.DoFn.BoundedPerElement, org.apache.beam.sdk.transforms.DoFn.BundleFinalizer, org.apache.beam.sdk.transforms.DoFn.Element, org.apache.beam.sdk.transforms.DoFn.FieldAccess, org.apache.beam.sdk.transforms.DoFn.FinishBundle, org.apache.beam.sdk.transforms.DoFn.FinishBundleContext, org.apache.beam.sdk.transforms.DoFn.GetInitialRestriction, org.apache.beam.sdk.transforms.DoFn.GetInitialWatermarkEstimatorState, org.apache.beam.sdk.transforms.DoFn.GetRestrictionCoder, org.apache.beam.sdk.transforms.DoFn.GetSize, org.apache.beam.sdk.transforms.DoFn.GetWatermarkEstimatorStateCoder, org.apache.beam.sdk.transforms.DoFn.Key, org.apache.beam.sdk.transforms.DoFn.MultiOutputReceiver, org.apache.beam.sdk.transforms.DoFn.NewTracker, org.apache.beam.sdk.transforms.DoFn.NewWatermarkEstimator, org.apache.beam.sdk.transforms.DoFn.OnTimer, org.apache.beam.sdk.transforms.DoFn.OnTimerContext, org.apache.beam.sdk.transforms.DoFn.OnTimerFamily, org.apache.beam.sdk.transforms.DoFn.OnWindowExpiration, org.apache.beam.sdk.transforms.DoFn.OnWindowExpirationContext, org.apache.beam.sdk.transforms.DoFn.OutputReceiver<T extends java.lang.Object>, org.apache.beam.sdk.transforms.DoFn.ProcessContext, org.apache.beam.sdk.transforms.DoFn.ProcessContinuation, org.apache.beam.sdk.transforms.DoFn.ProcessElement, org.apache.beam.sdk.transforms.DoFn.RequiresStableInput, org.apache.beam.sdk.transforms.DoFn.RequiresTimeSortedInput, org.apache.beam.sdk.transforms.DoFn.Restriction, org.apache.beam.sdk.transforms.DoFn.Setup, org.apache.beam.sdk.transforms.DoFn.SideInput, org.apache.beam.sdk.transforms.DoFn.SplitRestriction, org.apache.beam.sdk.transforms.DoFn.StartBundle, org.apache.beam.sdk.transforms.DoFn.StartBundleContext, org.apache.beam.sdk.transforms.DoFn.StateId, org.apache.beam.sdk.transforms.DoFn.Teardown, org.apache.beam.sdk.transforms.DoFn.TimerFamily, org.apache.beam.sdk.transforms.DoFn.TimerId, org.apache.beam.sdk.transforms.DoFn.Timestamp, org.apache.beam.sdk.transforms.DoFn.TruncateRestriction, org.apache.beam.sdk.transforms.DoFn.UnboundedPerElement, org.apache.beam.sdk.transforms.DoFn.WatermarkEstimatorState, org.apache.beam.sdk.transforms.DoFn.WindowedContext

Constructor Summary

Constructors

Constructor	Description
PostProcessingMetricsDoFn(ChangeStreamMetrics metrics)

Method Summary

Modifier and Type	Method	Description
void	processElement(DataChangeRecord dataChangeRecord, org.apache.beam.sdk.transforms.DoFn.OutputReceiver<DataChangeRecord> receiver)	Stage to measure a data records latencies and metrics.

Methods inherited from class org.apache.beam.sdk.transforms.DoFn

getAllowedTimestampSkew, getInputTypeDescriptor, getOutputTypeDescriptor, populateDisplayData, prepareForProcessing

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

PostProcessingMetricsDoFn

public PostProcessingMetricsDoFn(ChangeStreamMetrics metrics)

Method Detail

processElement

@ProcessElement
public void processElement(@Element
                           DataChangeRecord dataChangeRecord,
                           org.apache.beam.sdk.transforms.DoFn.OutputReceiver<DataChangeRecord> receiver)

Stage to measure a data records latencies and metrics. The metrics gathered are:

1. The number of data records emitted.
2. The latency between a record's Cloud Spanner commit timestamp and the time it reached this component (referred as emit timestamp).
3. The streaming latency of a record from the change stream query.

After measurement the record is re-emitted to the next stage.

Parameters:

dataChangeRecord - the record to gather metrics for

receiver - the output receiver of the PostProcessingMetricsDoFn SDF