public abstract static class PubsubIO.Write<T>
extends org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PCollection<T>,org.apache.beam.sdk.values.PDone>
Modifier and Type | Class and Description |
---|---|
class |
PubsubIO.Write.PubsubBoundedWriter
Writer to Pubsub which batches messages from bounded collections.
|
Constructor and Description |
---|
Write() |
Modifier and Type | Method and Description |
---|---|
org.apache.beam.sdk.values.PDone |
expand(org.apache.beam.sdk.values.PCollection<T> input) |
void |
populateDisplayData(org.apache.beam.sdk.transforms.display.DisplayData.Builder builder) |
PubsubIO.Write<T> |
to(org.apache.beam.sdk.transforms.SerializableFunction<org.apache.beam.sdk.values.ValueInSingleWindow<T>,java.lang.String> topicFunction)
Provides a function to dynamically specify the target topic per message.
|
PubsubIO.Write<T> |
to(java.lang.String topic)
Publishes to the specified topic.
|
PubsubIO.Write<T> |
to(org.apache.beam.sdk.options.ValueProvider<java.lang.String> topic)
Like
topic() but with a ValueProvider . |
PubsubIO.Write<T> |
withClientFactory(PubsubClient.PubsubClientFactory factory)
The default client to write to Pub/Sub is the
PubsubJsonClient , created by the PubsubJsonClient.PubsubJsonClientFactory . |
PubsubIO.Write<T> |
withErrorHandler(org.apache.beam.sdk.transforms.errorhandling.ErrorHandler<org.apache.beam.sdk.transforms.errorhandling.BadRecord,?> badRecordErrorHandler)
Writes any serialization failures out to the Error Handler.
|
PubsubIO.Write<T> |
withIdAttribute(java.lang.String idAttribute)
Writes to Pub/Sub, adding each record's unique identifier to the published messages in an
attribute with the specified name.
|
PubsubIO.Write<T> |
withMaxBatchBytesSize(int maxBatchBytesSize)
Writes to Pub/Sub are limited by 10mb in general.
|
PubsubIO.Write<T> |
withMaxBatchSize(int batchSize)
Writes to Pub/Sub are batched to efficiently send data.
|
PubsubIO.Write<T> |
withPubsubRootUrl(java.lang.String pubsubRootUrl) |
PubsubIO.Write<T> |
withTimestampAttribute(java.lang.String timestampAttribute)
Writes to Pub/Sub and adds each record's timestamp to the published messages in an attribute
with the specified name.
|
addAnnotation, compose, compose, getAdditionalInputs, getAnnotations, getDefaultOutputCoder, getDefaultOutputCoder, getDefaultOutputCoder, getKindString, getName, getResourceHints, setDisplayData, setResourceHints, toString, validate, validate
public PubsubIO.Write<T> to(java.lang.String topic)
See PubsubIO.PubsubTopic.fromPath(String)
for more details on the format of the
topic
string.
public PubsubIO.Write<T> to(org.apache.beam.sdk.options.ValueProvider<java.lang.String> topic)
topic()
but with a ValueProvider
.public PubsubIO.Write<T> to(org.apache.beam.sdk.transforms.SerializableFunction<org.apache.beam.sdk.values.ValueInSingleWindow<T>,java.lang.String> topicFunction)
to(java.lang.String)
is called again specifying a topic, then this
topicFunction will be ignored.public PubsubIO.Write<T> withClientFactory(PubsubClient.PubsubClientFactory factory)
PubsubJsonClient
, created by the PubsubJsonClient.PubsubJsonClientFactory
. This function allows to change the Pub/Sub client
by providing another PubsubClient.PubsubClientFactory
like the PubsubGrpcClientFactory
.public PubsubIO.Write<T> withMaxBatchSize(int batchSize)
Pub/Sub has a limitation of 10mb per individual request/batch. This attribute was requested dynamic to allow larger Pub/Sub messages to be sent using this source. Thus allowing customizable batches and control of number of events before the 10mb size limit is hit.
public PubsubIO.Write<T> withMaxBatchBytesSize(int maxBatchBytesSize)
public PubsubIO.Write<T> withTimestampAttribute(java.lang.String timestampAttribute)
Instant(long)
can be used to parse this value.
If the output from this sink is being read by another Beam pipeline, then PubsubIO.Read.withTimestampAttribute(String)
can be used to ensure the other source reads
these timestamps from the appropriate attribute.
public PubsubIO.Write<T> withIdAttribute(java.lang.String idAttribute)
If the output from this sink is being read by another Beam pipeline, then PubsubIO.Read.withIdAttribute(String)
can be used to ensure that* the other source reads
these unique identifiers from the appropriate attribute.
public PubsubIO.Write<T> withPubsubRootUrl(java.lang.String pubsubRootUrl)
public PubsubIO.Write<T> withErrorHandler(org.apache.beam.sdk.transforms.errorhandling.ErrorHandler<org.apache.beam.sdk.transforms.errorhandling.BadRecord,?> badRecordErrorHandler)
ErrorHandler
for
details on how to configure an Error Handler. Error Handlers are not well supported when
writing to topics with schemas, and it is not recommended to configure an error handler if
the target topic has a schema.public org.apache.beam.sdk.values.PDone expand(org.apache.beam.sdk.values.PCollection<T> input)
expand
in class org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PCollection<T>,org.apache.beam.sdk.values.PDone>
public void populateDisplayData(org.apache.beam.sdk.transforms.display.DisplayData.Builder builder)
populateDisplayData
in interface org.apache.beam.sdk.transforms.display.HasDisplayData
populateDisplayData
in class org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PCollection<T>,org.apache.beam.sdk.values.PDone>