Package org.apache.beam.sdk.io.gcp.spanner.changestreams.mapper

Class ChangeStreamRecordMapper

java.lang.Object

org.apache.beam.sdk.io.gcp.spanner.changestreams.mapper.ChangeStreamRecordMapper

public class ChangeStreamRecordMapper
extends java.lang.Object

This class is responsible for transforming a Struct to a List of ChangeStreamRecord models.

The change stream full specification can be seen in the internal documentation https://docs.google.com/document/d/1nLlMGvQLIeUSDNmtoLT9vaQo0hVGl4CIf6iCSOkdIbA/edit#bookmark=id.fxgtygh8eony

Method Summary

Modifier and Type	Method	Description
java.util.List<ChangeStreamRecord>	toChangeStreamRecords(PartitionMetadata partition, ChangeStreamResultSet resultSet, ChangeStreamResultSetMetadata resultSetMetadata)	In GoogleSQL, change stream records are returned as an array of Struct.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

toChangeStreamRecords

public java.util.List<ChangeStreamRecord> toChangeStreamRecords(PartitionMetadata partition,
                                                                ChangeStreamResultSet resultSet,
                                                                ChangeStreamResultSetMetadata resultSetMetadata)

In GoogleSQL, change stream records are returned as an array of Struct. In PostgresQL, change stream records are returned as Jsonb Transforms a / Jsonb representing a change stream result into a List of ChangeStreamRecord model. The type of the change stream record will be identified and one of the following subclasses can be returned within the resulting List:

• DataChangeRecord
• HeartbeatRecord
• ChildPartitionsRecord

Additionally to the / Jsonb received, the originating partition of the records (given by the PartitionMetadata parameter) and the stream metadata (given by the ChangeStreamResultSetMetadata) are used to populate the ChangeStreamRecordMetadata for each record mapped.

The / Jsonb is expected to have the following fields:

• DATA_CHANGE_RECORD_COLUMN: non-nullable Struct list of data change records or a Jsonb representing a child partition record.
  • COMMIT_TIMESTAMP_COLUMN: non-nullable Timestamp representing the timestamp at which the modifications within the record were committed in Cloud Spanner.
  • SERVER_TRANSACTION_ID_COLUMN: non-nullable String representing the unique transaction id in which the modifications for this record occurred.
  • IS_LAST_RECORD_IN_TRANSACTION_IN_PARTITION_COLUMN: non-nullable Boolean indicating whether this record is the last emitted for the transaction.
  • RECORD_SEQUENCE_COLUMN: non-nullable String representing the order in which this record appears within the context of a partition, commit timestamp and transaction.
  • TABLE_NAME_COLUMN: non-nullable String representing the name of the table in which the modifications for this record occurred.
  • COLUMN_TYPES_COLUMN: non-nullable List of Structs representing the type of the primary keys and modified columns within this record.
    • NAME_COLUMN: non-nullable String representing the name of a column.
    • TYPE_COLUMN: non-nullable String representing the type of a column.
    • IS_PRIMARY_KEY_COLUMN: non-nullable Boolean indicating if the column is part of the primary key.
    • ORDINAL_POSITION_COLUMN: non-nullable Long representing the position of the column in the table it is defined.
  • MODS_COLUMN: non-nullable List of Structs representing the data modifications within this record.
    • KEYS_COLUMN: non-nullable String json object, where keys are the primary key column names, and the values are their corresponding values.
    • OLD_VALUES_COLUMN: nullable String json object displaying the old state of the columns modified, where keys are the column names, and the values are their corresponding values.
    • NEW_VALUES_COLUMN: nullable String json object displaying the new state of the columns modified, where keys are the column names, and the values are their corresponding values.
  • MOD_TYPE_COLUMN: non-nullable String representing the type of operation that caused the modifications (see also ModType.
  • VALUE_CAPTURE_TYPE_COLUMN: non-nullable String representing the capture type of the change stream that generated this record (see also ValueCaptureType).
  • NUMBER_OF_RECORDS_IN_TRANSACTION_COLUMN: non-nullable Long representing the total number of data change records for the transaction in which this record occurred.
  • NUMBER_OF_PARTITIONS_IN_TRANSACTION_COLUMN: non-nullable Long representing the total number of partitions for the transaction in which this record occurred.
• HEARTBEAT_RECORD_COLUMN: non-nullable Struct list of hearbeat records or a Jsonb representing a child partition record.
  • TIMESTAMP_COLUMN: non-nullable Timestamp representing the timestamp for which the change stream query has returned all changes (see more in HeartbeatRecord.getTimestamp().
• CHILD_PARTITIONS_RECORD_COLUMN: non-nullable Struct list of child partitions records or a Jsonb representing a child partition record.
  • START_TIMESTAMP_COLUMN: non-nullable Timestamp representing the timestamp at which the new partition started being valid in Cloud Spanner.
  • RECORD_SEQUENCE_COLUMN: non-nullable String representing the order in which this record appears within the context of a partition and commit timestamp.
  • CHILD_PARTITIONS_COLUMN: non-nullable List of Struct representing the new child partitions.
    • TOKEN_COLUMN: non-nullable String representing the unique identifier of the new child partition.
    • PARENT_PARTITION_TOKENS_COLUMN: non-nullable List of String representing the unique identifier(s) of parent partition(s) where this child partition originated from.

Parameters:

partition - the partition metadata from which the row was generated

resultSet - the change stream result set

resultSetMetadata - the metadata generated when reading the change stream row

Returns:

a List of ChangeStreamRecord subclasses