Class PubsubIO.Write<T>
- java.lang.Object
-
- org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PCollection<T>,org.apache.beam.sdk.values.PDone>
-
- org.apache.beam.sdk.io.gcp.pubsub.PubsubIO.Write<T>
-
- All Implemented Interfaces:
java.io.Serializable
,org.apache.beam.sdk.transforms.display.HasDisplayData
- Enclosing class:
- PubsubIO
public abstract static class PubsubIO.Write<T> extends org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PCollection<T>,org.apache.beam.sdk.values.PDone>
Implementation of write methods.- See Also:
- Serialized Form
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description class
PubsubIO.Write.PubsubBoundedWriter
Writer to Pubsub which batches messages from bounded collections.
-
Constructor Summary
Constructors Constructor Description Write()
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description org.apache.beam.sdk.values.PDone
expand(org.apache.beam.sdk.values.PCollection<T> input)
void
populateDisplayData(org.apache.beam.sdk.transforms.display.DisplayData.Builder builder)
PubsubIO.Write<T>
to(java.lang.String topic)
Publishes to the specified topic.PubsubIO.Write<T>
to(org.apache.beam.sdk.options.ValueProvider<java.lang.String> topic)
Liketopic()
but with aValueProvider
.PubsubIO.Write<T>
withClientFactory(PubsubClient.PubsubClientFactory factory)
The default client to write to Pub/Sub is thePubsubJsonClient
, created by thePubsubJsonClient.PubsubJsonClientFactory
.PubsubIO.Write<T>
withIdAttribute(java.lang.String idAttribute)
Writes to Pub/Sub, adding each record's unique identifier to the published messages in an attribute with the specified name.PubsubIO.Write<T>
withMaxBatchBytesSize(int maxBatchBytesSize)
Writes to Pub/Sub are limited by 10mb in general.PubsubIO.Write<T>
withMaxBatchSize(int batchSize)
Writes to Pub/Sub are batched to efficiently send data.PubsubIO.Write<T>
withPubsubRootUrl(java.lang.String pubsubRootUrl)
PubsubIO.Write<T>
withTimestampAttribute(java.lang.String timestampAttribute)
Writes to Pub/Sub and adds each record's timestamp to the published messages in an attribute with the specified name.
-
-
-
Method Detail
-
to
public PubsubIO.Write<T> to(java.lang.String topic)
Publishes to the specified topic.See
PubsubIO.PubsubTopic.fromPath(String)
for more details on the format of thetopic
string.
-
to
public PubsubIO.Write<T> to(org.apache.beam.sdk.options.ValueProvider<java.lang.String> topic)
Liketopic()
but with aValueProvider
.
-
withClientFactory
public PubsubIO.Write<T> withClientFactory(PubsubClient.PubsubClientFactory factory)
The default client to write to Pub/Sub is thePubsubJsonClient
, created by thePubsubJsonClient.PubsubJsonClientFactory
. This function allows to change the Pub/Sub client by providing anotherPubsubClient.PubsubClientFactory
like thePubsubGrpcClientFactory
.
-
withMaxBatchSize
public PubsubIO.Write<T> withMaxBatchSize(int batchSize)
Writes to Pub/Sub are batched to efficiently send data. The value of the attribute will be a number representing the number of Pub/Sub messages to queue before sending off the bulk request. For example, if given 1000 the write sink will wait until 1000 messages have been received, or the pipeline has finished, whichever is first.Pub/Sub has a limitation of 10mb per individual request/batch. This attribute was requested dynamic to allow larger Pub/Sub messages to be sent using this source. Thus allowing customizable batches and control of number of events before the 10mb size limit is hit.
-
withMaxBatchBytesSize
public PubsubIO.Write<T> withMaxBatchBytesSize(int maxBatchBytesSize)
Writes to Pub/Sub are limited by 10mb in general. This attribute controls the maximum allowed bytes to be sent to Pub/Sub in a single batched message.
-
withTimestampAttribute
public PubsubIO.Write<T> withTimestampAttribute(java.lang.String timestampAttribute)
Writes to Pub/Sub and adds each record's timestamp to the published messages in an attribute with the specified name. The value of the attribute will be a number representing the number of milliseconds since the Unix epoch. For example, if using the Joda time classes,Instant(long)
can be used to parse this value.If the output from this sink is being read by another Beam pipeline, then
PubsubIO.Read.withTimestampAttribute(String)
can be used to ensure the other source reads these timestamps from the appropriate attribute.
-
withIdAttribute
public PubsubIO.Write<T> withIdAttribute(java.lang.String idAttribute)
Writes to Pub/Sub, adding each record's unique identifier to the published messages in an attribute with the specified name. The value of the attribute is an opaque string.If the output from this sink is being read by another Beam pipeline, then
PubsubIO.Read.withIdAttribute(String)
can be used to ensure that* the other source reads these unique identifiers from the appropriate attribute.
-
withPubsubRootUrl
public PubsubIO.Write<T> withPubsubRootUrl(java.lang.String pubsubRootUrl)
-
expand
public org.apache.beam.sdk.values.PDone expand(org.apache.beam.sdk.values.PCollection<T> input)
- Specified by:
expand
in classorg.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PCollection<T>,org.apache.beam.sdk.values.PDone>
-
populateDisplayData
public void populateDisplayData(org.apache.beam.sdk.transforms.display.DisplayData.Builder builder)
- Specified by:
populateDisplayData
in interfaceorg.apache.beam.sdk.transforms.display.HasDisplayData
- Overrides:
populateDisplayData
in classorg.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PCollection<T>,org.apache.beam.sdk.values.PDone>
-
-