Package org.apache.beam.sdk.io

Class AvroSource<T>

  • Type Parameters:
    T - The type of records to be read from the source.
    All Implemented Interfaces:
    java.io.Serializable, HasDisplayData
    @Experimental(SOURCE_SINK)
public class AvroSource<T>
extends BlockBasedSource<T>
    Do not use in pipelines directly: most users should use AvroIO.Read.

    A FileBasedSource for reading Avro files.

    To read a PCollection of objects from one or more Avro files, use from(org.apache.beam.sdk.options.ValueProvider<java.lang.String>) to specify the path(s) of the files to read. The AvroSource that is returned will read objects of type GenericRecord with the schema(s) that were written at file creation. To further configure the AvroSource to read with a user-defined schema, or to return records of a type other than GenericRecord, use withSchema(Schema) (using an Avro Schema), withSchema(String) (using a JSON schema), or withSchema(Class) (to return objects of the Avro-generated class specified).

    An AvroSource can be read from using the Read transform. For example: 

    
 AvroSource<MyType> source = AvroSource.from(file.toPath()).withSchema(MyType.class);
 PCollection<MyType> records = Read.from(mySource);

    This class's implementation is based on the Avro 1.7.7 specification and implements parsing of some parts of Avro Object Container Files. The rationale for doing so is that the Avro API does not provide efficient ways of computing the precise offsets of blocks within a file, which is necessary to support dynamic work rebalancing. However, whenever it is possible to use the Avro API in a way that supports maintaining precise offsets, this class uses the Avro API.

    Avro Object Container files store records in blocks. Each block contains a collection of records. Blocks may be encoded (e.g., with bzip2, deflate, snappy, etc.). Blocks are delineated from one another by a 16-byte sync marker.

    An AvroSource for a subrange of a single file contains records in the blocks such that the start offset of the block is greater than or equal to the start offset of the source and less than the end offset of the source.

    To use XZ-encoded Avro files, please include an explicit dependency on xz-1.8.jar, which has been marked as optional in the Maven sdk/pom.xml. 

    
 <dependency>
   <groupId>org.tukaani</groupId>
   <artifactId>xz</artifactId>
   <version>1.8</version>
 </dependency>

    Permissions

    Permission requirements depend on the PipelineRunner that is used to execute the pipeline. Please refer to the documentation of corresponding PipelineRunners for more details.

    See Also:
    Serialized Form

    • Method Detail

      • from

        public static AvroSource<org.apache.avro.generic.GenericRecord> from​(ValueProvider<java.lang.String> fileNameOrPattern)
        Reads from the given file name or pattern ("glob"). The returned source needs to be further configured by calling withSchema(java.lang.String) to return a type other than GenericRecord.

      • withSchema

        public AvroSource<org.apache.avro.generic.GenericRecord> withSchema​(java.lang.String schema)
        Reads files containing records that conform to the given schema.

      • withSchema

        public AvroSource<org.apache.avro.generic.GenericRecord> withSchema​(org.apache.avro.Schema schema)
        Like withSchema(String).

      • withSchema

        public <X> AvroSource<X> withSchema​(java.lang.Class<X> clazz)
        Reads files containing records of the given class.

      • withParseFn

        public <X> AvroSource<X> withParseFn​(SerializableFunction<org.apache.avro.generic.GenericRecord,​X> parseFn,
                                     Coder<X> coder)
        Reads GenericRecord of unspecified schema and maps them to instances of a custom type using the given parseFn and encoded using the given coder.

      • withMinBundleSize

        public AvroSource<T> withMinBundleSize​(long minBundleSize)
        Sets the minimum bundle size. Refer to OffsetBasedSource for a description of minBundleSize and its use.

      • validate

        public void validate()
        Description copied from class: Source
        Checks that this source is valid, before it can be used in a pipeline.

        It is recommended to use Preconditions for implementing this method.

        Overrides:
        validate in class FileBasedSource<T>

      • createForSubrangeOfFile

        @Deprecated
public BlockBasedSource<T> createForSubrangeOfFile​(java.lang.String fileName,
                                                   long start,
                                                   long end)
                                            throws java.io.IOException
        Deprecated.
        Used by Dataflow worker
        Used by the Dataflow worker. Do not introduce new usages. Do not delete without confirming that Dataflow ValidatesRunner tests pass.
        Throws:
        java.io.IOException

      • getOutputCoder

        public Coder<T> getOutputCoder()
        Description copied from class: Source
        Returns the Coder to use for the data read from this source.
        Overrides:
        getOutputCoder in class Source<T>