com.google.cloud.dataflow.sdk.coders

Class AvroCoder<T>

java.lang.Object

com.google.cloud.dataflow.sdk.coders.StandardCoder<T>

com.google.cloud.dataflow.sdk.coders.AvroCoder<T>

Type Parameters:

T - the type of elements handled by this coder

All Implemented Interfaces:

Coder<T>, java.io.Serializable

public class AvroCoder<T>
extends StandardCoder<T>

An encoder using Avro binary format.

The Avro schema is generated using reflection on the element type, using Avro's org.apache.avro.reflect.ReflectData, and encoded as part of the Coder instance.

For complete details about schema generation and how it can be controlled please see the org.apache.avro.reflect package. Only concrete classes with a no-argument constructor can be mapped to Avro records. All inherited fields that are not static or transient are used. Fields are not permitted to be null unless annotated by org.apache.avro.reflect.Nullable or a org.apache.avro.reflect.Union containing null.

To use, specify the Coder type on a PCollection:

 PCollection<MyCustomElement> records =
     input.apply(...)
          .setCoder(AvroCoder.of(MyCustomElement.class);
 
 

or annotate the element class using @DefaultCoder.

 @DefaultCoder(AvroCoder.class)
 public class MyCustomElement {
   ...
 }
 

The implementation attempts to determine if the Avro encoding of the given type will satisfy the criteria of Coder.verifyDeterministic() by inspecting both the type and the Schema provided or generated by Avro. Only coders that are deterministic can be used in GroupByKey operations.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected static class	AvroCoder.AvroDeterminismChecker Helper class encapsulating the various pieces of state maintained by the recursive walk used for checking if the encoding will be deterministic.

Nested classes/interfaces inherited from interface com.google.cloud.dataflow.sdk.coders.Coder

Coder.Context, Coder.NonDeterministicException

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	AvroCoder(java.lang.Class<T> type, org.apache.avro.Schema schema)

Method Summary

Methods

Modifier and Type	Method and Description
com.google.cloud.dataflow.sdk.util.CloudObject	asCloudObject() Returns the CloudObject that represents this Coder.
org.apache.avro.io.DatumReader<T>	createDatumReader() Returns a new DatumReader that can be used to read from an Avro file directly.
org.apache.avro.io.DatumWriter<T>	createDatumWriter() Returns a new DatumWriter that can be used to write to an Avro file directly.
T	decode(java.io.InputStream inStream, Coder.Context context) Decodes a value of type T from the given input stream in the given context.
void	encode(T value, java.io.OutputStream outStream, Coder.Context context) Encodes the given value of type T onto the given output stream in the given context.
java.util.List<? extends Coder<?>>	getCoderArguments() If this is a Coder for a parameterized type, returns the list of Coders being used for each of the parameters, or returns null if this cannot be done or this is not a parameterized type.
org.apache.avro.Schema	getSchema() Returns the schema used by this coder.
boolean	isDeterministic() Deprecated.
static <T> AvroCoder<T>	of(java.lang.Class<T> type) Returns an AvroCoder instance for the provided element type.
static <T> AvroCoder<T>	of(java.lang.Class<T> type, org.apache.avro.Schema schema) Returns an AvroCoder instance for the provided element type using the provided Avro schema.
static AvroCoder<org.apache.avro.generic.GenericRecord>	of(org.apache.avro.Schema schema) Returns an AvroCoder instance for the Avro schema.
static AvroCoder<?>	of(java.lang.String classType, java.lang.String schema)
void	verifyDeterministic() Raises an exception describing reasons why the type may not be deterministically encoded using the given Schema, the directBinaryEncoder, and the ReflectDatumWriter or GenericDatumWriter.

Methods inherited from class com.google.cloud.dataflow.sdk.coders.StandardCoder

equals, getComponents, getEncodedElementByteSize, hashCode, isRegisterByteSizeObserverCheap, registerByteSizeObserver, toString, verifyDeterministic, verifyDeterministic

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Constructor Detail

AvroCoder

protected AvroCoder(java.lang.Class<T> type,
         org.apache.avro.Schema schema)

Method Detail

of

public static <T> AvroCoder<T> of(java.lang.Class<T> type)

Returns an AvroCoder instance for the provided element type.

Type Parameters:

T - the element type

of

public static AvroCoder<org.apache.avro.generic.GenericRecord> of(org.apache.avro.Schema schema)

Returns an AvroCoder instance for the Avro schema. The implicit type is GenericRecord.

of

public static <T> AvroCoder<T> of(java.lang.Class<T> type,
                  org.apache.avro.Schema schema)

Returns an AvroCoder instance for the provided element type using the provided Avro schema.

If the type argument is GenericRecord, the schema may be arbitrary. Otherwise, the schema must correspond to the type provided.

Type Parameters:

T - the element type

of

public static AvroCoder<?> of(java.lang.String classType,
              java.lang.String schema)
                       throws java.lang.ClassNotFoundException

Throws:

java.lang.ClassNotFoundException

encode

public void encode(T value,
          java.io.OutputStream outStream,
          Coder.Context context)
            throws java.io.IOException

Description copied from interface: Coder

Encodes the given value of type T onto the given output stream in the given context.

Throws:

java.io.IOException - if writing to the OutputStream fails for some reason

CoderException - if the value could not be encoded for some reason

decode

public T decode(java.io.InputStream inStream,
       Coder.Context context)
         throws java.io.IOException

Description copied from interface: Coder

Decodes a value of type T from the given input stream in the given context. Returns the decoded value.

Throws:

java.io.IOException - if reading from the InputStream fails for some reason

CoderException - if the value could not be decoded for some reason

getCoderArguments

public java.util.List<? extends Coder<?>> getCoderArguments()

Description copied from interface: Coder

If this is a Coder for a parameterized type, returns the list of Coders being used for each of the parameters, or returns null if this cannot be done or this is not a parameterized type.

asCloudObject

public com.google.cloud.dataflow.sdk.util.CloudObject asCloudObject()

Description copied from interface: Coder

Returns the CloudObject that represents this Coder.

Specified by:

asCloudObject in interface Coder<T>

Overrides:

asCloudObject in class StandardCoder<T>

isDeterministic

@Deprecated
public boolean isDeterministic()

Deprecated.

Returns true if the given type should be deterministically encoded using the given Schema, the directBinaryEncoder, and the ReflectDatumWriter or GenericDatumWriter.

verifyDeterministic

public void verifyDeterministic()
                         throws Coder.NonDeterministicException

Raises an exception describing reasons why the type may not be deterministically encoded using the given Schema, the directBinaryEncoder, and the ReflectDatumWriter or GenericDatumWriter.

Specified by:

verifyDeterministic in interface Coder<T>

Overrides:

verifyDeterministic in class StandardCoder<T>

Throws:

Coder.NonDeterministicException - if this coder is not deterministic.

createDatumReader

public org.apache.avro.io.DatumReader<T> createDatumReader()

Returns a new DatumReader that can be used to read from an Avro file directly.

createDatumWriter

public org.apache.avro.io.DatumWriter<T> createDatumWriter()

Returns a new DatumWriter that can be used to write to an Avro file directly.

getSchema

public org.apache.avro.Schema getSchema()

Returns the schema used by this coder.