org.apache.beam.sdk.io.kafka

Class KafkaIO.Write<K,V>

java.lang.Object

org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PCollection<org.apache.beam.sdk.values.KV<K,V>>,org.apache.beam.sdk.values.PDone>

org.apache.beam.sdk.io.kafka.KafkaIO.Write<K,V>

All Implemented Interfaces:

java.io.Serializable, org.apache.beam.sdk.transforms.display.HasDisplayData

Enclosing class:

KafkaIO

public abstract static class KafkaIO.Write<K,V>
extends org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PCollection<org.apache.beam.sdk.values.KV<K,V>>,org.apache.beam.sdk.values.PDone>

A PTransform to write to a Kafka topic with KVs . See KafkaIO for more information on usage and configuration.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	KafkaIO.Write.External Exposes KafkaIO.Write as an external transform for cross-language usage.

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.Class<org.apache.beam.sdk.io.kafka.AutoValue_KafkaIO_Write>	AUTOVALUE_CLASS

Fields inherited from class org.apache.beam.sdk.transforms.PTransform

annotations, displayData, name, resourceHints

Constructor Summary

Constructors

Constructor and Description
Write()

Method Summary

Modifier and Type	Method and Description
org.apache.beam.sdk.values.PDone	expand(org.apache.beam.sdk.values.PCollection<org.apache.beam.sdk.values.KV<K,V>> input)
abstract KafkaIO.WriteRecords<K,V>	getWriteRecordsTransform()
void	populateDisplayData(org.apache.beam.sdk.transforms.display.DisplayData.Builder builder)
KafkaIO.Write<K,V>	updateProducerProperties(java.util.Map<java.lang.String,java.lang.Object> configUpdates) Deprecated. as of version 2.13. Use withProducerConfigUpdates(Map) instead.
void	validate(@Nullable org.apache.beam.sdk.options.PipelineOptions options)
org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PCollection<V>,org.apache.beam.sdk.values.PDone>	values() Writes just the values to Kafka.
KafkaIO.Write<K,V>	withBadRecordErrorHandler(org.apache.beam.sdk.transforms.errorhandling.ErrorHandler<org.apache.beam.sdk.transforms.errorhandling.BadRecord,?> badRecordErrorHandler) Configure a ErrorHandler.BadRecordErrorHandler for sending records to if they fail to serialize when being sent to Kafka.
KafkaIO.Write<K,V>	withBootstrapServers(java.lang.String bootstrapServers) Wrapper method over KafkaIO.WriteRecords.withBootstrapServers(String), used to keep the compatibility with old API based on KV type of element.
KafkaIO.Write<K,V>	withConsumerFactoryFn(org.apache.beam.sdk.transforms.SerializableFunction<java.util.Map<java.lang.String,java.lang.Object>,? extends org.apache.kafka.clients.consumer.Consumer<?,?>> consumerFactoryFn) Wrapper method over KafkaIO.WriteRecords.withConsumerFactoryFn(SerializableFunction), used to keep the compatibility with old API based on KV type of element.
KafkaIO.Write<K,V>	withEOS(int numShards, java.lang.String sinkGroupId) Wrapper method over KafkaIO.WriteRecords.withEOS(int, String), used to keep the compatibility with old API based on KV type of element.
KafkaIO.Write<K,V>	withInputTimestamp() Wrapper method over KafkaIO.WriteRecords.withInputTimestamp(), used to keep the compatibility with old API based on KV type of element.
KafkaIO.Write<K,V>	withKeySerializer(java.lang.Class<? extends org.apache.kafka.common.serialization.Serializer<K>> keySerializer) Wrapper method over KafkaIO.WriteRecords.withKeySerializer(Class), used to keep the compatibility with old API based on KV type of element.
KafkaIO.Write<K,V>	withProducerConfigUpdates(java.util.Map<java.lang.String,java.lang.Object> configUpdates) Update configuration for the producer.
KafkaIO.Write<K,V>	withProducerFactoryFn(org.apache.beam.sdk.transforms.SerializableFunction<java.util.Map<java.lang.String,java.lang.Object>,org.apache.kafka.clients.producer.Producer<K,V>> producerFactoryFn) Wrapper method over KafkaIO.WriteRecords.withProducerFactoryFn(SerializableFunction), used to keep the compatibility with old API based on KV type of element.
KafkaIO.Write<K,V>	withPublishTimestampFunction(KafkaPublishTimestampFunction<org.apache.beam.sdk.values.KV<K,V>> timestampFunction) Deprecated. use KafkaIO.WriteRecords and ProducerRecords to set publish timestamp.
KafkaIO.Write<K,V>	withTopic(java.lang.String topic) Wrapper method over KafkaIO.WriteRecords.withTopic(String), used to keep the compatibility with old API based on KV type of element.
KafkaIO.Write<K,V>	withValueSerializer(java.lang.Class<? extends org.apache.kafka.common.serialization.Serializer<V>> valueSerializer) Wrapper method over KafkaIO.WriteRecords.withValueSerializer(Class), used to keep the compatibility with old API based on KV type of element.

Methods inherited from class org.apache.beam.sdk.transforms.PTransform

addAnnotation, compose, compose, getAdditionalInputs, getAnnotations, getDefaultOutputCoder, getDefaultOutputCoder, getDefaultOutputCoder, getKindString, getName, getResourceHints, setDisplayData, setResourceHints, toString, validate

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

AUTOVALUE_CLASS

public static final java.lang.Class<org.apache.beam.sdk.io.kafka.AutoValue_KafkaIO_Write> AUTOVALUE_CLASS

Constructor Detail

Write

public Write()

Method Detail

getWriteRecordsTransform

public abstract KafkaIO.WriteRecords<K,V> getWriteRecordsTransform()

withBootstrapServers

public KafkaIO.Write<K,V> withBootstrapServers(java.lang.String bootstrapServers)

Wrapper method over KafkaIO.WriteRecords.withBootstrapServers(String), used to keep the compatibility with old API based on KV type of element.

withTopic

public KafkaIO.Write<K,V> withTopic(java.lang.String topic)

Wrapper method over KafkaIO.WriteRecords.withTopic(String), used to keep the compatibility with old API based on KV type of element.

withKeySerializer

public KafkaIO.Write<K,V> withKeySerializer(java.lang.Class<? extends org.apache.kafka.common.serialization.Serializer<K>> keySerializer)

Wrapper method over KafkaIO.WriteRecords.withKeySerializer(Class), used to keep the compatibility with old API based on KV type of element.

withValueSerializer

public KafkaIO.Write<K,V> withValueSerializer(java.lang.Class<? extends org.apache.kafka.common.serialization.Serializer<V>> valueSerializer)

Wrapper method over KafkaIO.WriteRecords.withValueSerializer(Class), used to keep the compatibility with old API based on KV type of element.

withProducerFactoryFn

public KafkaIO.Write<K,V> withProducerFactoryFn(org.apache.beam.sdk.transforms.SerializableFunction<java.util.Map<java.lang.String,java.lang.Object>,org.apache.kafka.clients.producer.Producer<K,V>> producerFactoryFn)

Wrapper method over KafkaIO.WriteRecords.withProducerFactoryFn(SerializableFunction), used to keep the compatibility with old API based on KV type of element.

withInputTimestamp

public KafkaIO.Write<K,V> withInputTimestamp()

Wrapper method over KafkaIO.WriteRecords.withInputTimestamp(), used to keep the compatibility with old API based on KV type of element.

withPublishTimestampFunction

@Deprecated
public KafkaIO.Write<K,V> withPublishTimestampFunction(KafkaPublishTimestampFunction<org.apache.beam.sdk.values.KV<K,V>> timestampFunction)

Deprecated. use KafkaIO.WriteRecords and ProducerRecords to set publish timestamp.

Wrapper method over KafkaIO.WriteRecords.withPublishTimestampFunction(KafkaPublishTimestampFunction), used to keep the compatibility with old API based on KV type of element.

withEOS

public KafkaIO.Write<K,V> withEOS(int numShards,
                                  java.lang.String sinkGroupId)

Wrapper method over KafkaIO.WriteRecords.withEOS(int, String), used to keep the compatibility with old API based on KV type of element.

withConsumerFactoryFn

public KafkaIO.Write<K,V> withConsumerFactoryFn(org.apache.beam.sdk.transforms.SerializableFunction<java.util.Map<java.lang.String,java.lang.Object>,? extends org.apache.kafka.clients.consumer.Consumer<?,?>> consumerFactoryFn)

Wrapper method over KafkaIO.WriteRecords.withConsumerFactoryFn(SerializableFunction), used to keep the compatibility with old API based on KV type of element.

updateProducerProperties

@Deprecated
public KafkaIO.Write<K,V> updateProducerProperties(java.util.Map<java.lang.String,java.lang.Object> configUpdates)

Deprecated. as of version 2.13. Use withProducerConfigUpdates(Map) instead.

Adds the given producer properties, overriding old values of properties with the same key.

withProducerConfigUpdates

public KafkaIO.Write<K,V> withProducerConfigUpdates(java.util.Map<java.lang.String,java.lang.Object> configUpdates)

Update configuration for the producer. Note that the default producer properties will not be completely overridden. This method only updates the value which has the same key.

By default, the producer uses the configuration from KafkaIO.WriteRecords.DEFAULT_PRODUCER_PROPERTIES.

withBadRecordErrorHandler

public KafkaIO.Write<K,V> withBadRecordErrorHandler(org.apache.beam.sdk.transforms.errorhandling.ErrorHandler<org.apache.beam.sdk.transforms.errorhandling.BadRecord,?> badRecordErrorHandler)

Configure a ErrorHandler.BadRecordErrorHandler for sending records to if they fail to serialize when being sent to Kafka.

expand

public org.apache.beam.sdk.values.PDone expand(org.apache.beam.sdk.values.PCollection<org.apache.beam.sdk.values.KV<K,V>> input)

Specified by:

expand in class org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PCollection<org.apache.beam.sdk.values.KV<K,V>>,org.apache.beam.sdk.values.PDone>

validate

public void validate(@Nullable org.apache.beam.sdk.options.PipelineOptions options)

Overrides:

validate in class org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PCollection<org.apache.beam.sdk.values.KV<K,V>>,org.apache.beam.sdk.values.PDone>

populateDisplayData

public void populateDisplayData(org.apache.beam.sdk.transforms.display.DisplayData.Builder builder)

Specified by:

populateDisplayData in interface org.apache.beam.sdk.transforms.display.HasDisplayData

Overrides:

populateDisplayData in class org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PCollection<org.apache.beam.sdk.values.KV<K,V>>,org.apache.beam.sdk.values.PDone>

values

public org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PCollection<V>,org.apache.beam.sdk.values.PDone> values()

Writes just the values to Kafka. This is useful for writing collections of values rather thank KVs.