com.google.cloud.dataflow.sdk.io

Class Source.Reader<T>

java.lang.Object

com.google.cloud.dataflow.sdk.io.Source.Reader<T>

All Implemented Interfaces:

AutoCloseable

Direct Known Subclasses:

BoundedSource.BoundedReader, UnboundedSource.UnboundedReader

Enclosing class:

Source<T>

public abstract static class Source.Reader<T>
extends Object
implements AutoCloseable

The interface that readers of custom input sources must implement.

This interface is deliberately distinct from Iterator because the current model tends to be easier to program and more efficient in practice for iterating over sources such as files, databases etc. (rather than pure collections).

Reader implementations do not need to be thread-safe; they may only be accessed by a single thread at once.

Callers of Readers must obey the following access pattern:

• One call to start()
  • If start() returned true, any number of calls to getCurrent* methods
• Repeatedly, a call to advance(). This may be called regardless of what the previous start()/advance() returned.
  • If advance() returned true, any number of calls to getCurrent* methods

For example, if the reader is reading a fixed set of data:

 for (boolean available = reader.start(); available; available = reader.advance()) {
   T item = reader.getCurrent();
   Instant timestamp = reader.getCurrentTimestamp();
   ...
 }
 

If the set of data being read is continually growing:

 boolean available = reader.start();
 while (true) {
   if (available) {
     T item = reader.getCurrent();
     Instant timestamp = reader.getCurrentTimestamp();
     ...
     resetExponentialBackoff();
   } else {
     exponentialBackoff();
   }
   available = reader.advance();
 }
 

Note: this interface is a work-in-progress and may change.

Constructor Summary

Constructors

Constructor and Description
Reader()

Method Summary

Modifier and Type	Method and Description
abstract boolean	advance() Advances the reader to the next valid record.
abstract void	close() Closes the reader.
abstract T	getCurrent() Returns the value of the data item that was read by the last start() or advance() call.
abstract Source<T>	getCurrentSource() Returns a Source describing the same input that this Reader reads (including items already read).
abstract Instant	getCurrentTimestamp() Returns the timestamp associated with the current data item.
abstract boolean	start() Initializes the reader and advances the reader to the first record.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Reader

public Reader()

Method Detail

start

public abstract boolean start()
                       throws IOException

Initializes the reader and advances the reader to the first record.

This method should be called exactly once. The invocation should occur prior to calling advance() or getCurrent(). This method may perform expensive operations that are needed to initialize the reader.

Returns:

true if a record was read, false if there is no more input available.

Throws:

IOException

advance

public abstract boolean advance()
                         throws IOException

Advances the reader to the next valid record.

Returns:

true if a record was read, false if there is no more input available.

Throws:

IOException

getCurrent

public abstract T getCurrent()
                      throws NoSuchElementException

Returns the value of the data item that was read by the last start() or advance() call. The returned value must be effectively immutable and remain valid indefinitely.

Multiple calls to this method without an intervening call to advance() should return the same result.

Throws:

NoSuchElementException - if the reader is at the beginning of the input and start() or advance() wasn't called, or if the last start() or advance() returned false.

getCurrentTimestamp

public abstract Instant getCurrentTimestamp()
                                     throws NoSuchElementException

Returns the timestamp associated with the current data item.

If the source does not support timestamps, this should return BoundedWindow.TIMESTAMP_MIN_VALUE.

Multiple calls to this method without an intervening call to advance() should return the same result.

Throws:

NoSuchElementException - if the reader is at the beginning of the input and start() or advance() wasn't called, or if the last start() or advance() returned false.

close

public abstract void close()
                    throws IOException

Closes the reader. The reader cannot be used after this method is called.

Specified by:

close in interface AutoCloseable

Throws:

IOException

getCurrentSource

public abstract Source<T> getCurrentSource()

Returns a Source describing the same input that this Reader reads (including items already read).

A reader created from the result of getCurrentSource, if consumed, MUST return the same data items as the current reader.