com.google.cloud.dataflow.sdk.util

Class StreamingModeExecutionContext

java.lang.Object

com.google.cloud.dataflow.sdk.util.BaseExecutionContext

com.google.cloud.dataflow.sdk.util.BatchModeExecutionContext

com.google.cloud.dataflow.sdk.runners.worker.DataflowExecutionContext

com.google.cloud.dataflow.sdk.util.StreamingModeExecutionContext

All Implemented Interfaces:

ExecutionContext

public class StreamingModeExecutionContext
extends DataflowExecutionContext

ExecutionContext for use in streaming mode.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	StreamingModeExecutionContext.StreamingModeSideInputReader A SideInputReader that fetches side inputs from the streaming worker's cache.

Constructor Summary

Constructors

Constructor and Description
StreamingModeExecutionContext(StateFetcher stateFetcher, ConcurrentMap<com.google.protobuf.ByteString,UnboundedSource.UnboundedReader<?>> readerCache, ConcurrentMap<String,String> stateNameMap)

Method Summary

Modifier and Type	Method and Description
ExecutionContext.StepContext	createStepContext(String stepName, String transformName) Create a new ExecutionContext.StepContext.
Map<Long,Runnable>	flushState()
UnboundedSource.UnboundedReader<?>	getCachedReader()
com.google.cloud.dataflow.sdk.runners.worker.windmill.Windmill.WorkItemCommitRequest.Builder	getOutputBuilder()
UnboundedSource.CheckpointMark	getReaderCheckpoint(Coder<? extends UnboundedSource.CheckpointMark> coder)
List<Long>	getReadyCommitCallbackIds()
com.google.protobuf.ByteString	getSerializedKey()
Iterable<com.google.cloud.dataflow.sdk.runners.worker.windmill.Windmill.GlobalDataId>	getSideInputNotifications()
SideInputReader	getSideInputReader(Iterable<? extends com.google.api.services.dataflow.model.SideInputInfo> sideInputInfos) Returns a SideInputReader for all the side inputs described in the given SideInputInfo descriptors.
SideInputReader	getSideInputReaderForViews(Iterable<? extends PCollectionView<?>> views) Returns a SideInputReader for all the provided views, where the execution context itself knows how to read data for the view.
com.google.cloud.dataflow.sdk.runners.worker.windmill.Windmill.WorkItem	getWork()
long	getWorkToken()
void	setActiveReader(UnboundedSource.UnboundedReader<?> reader)
void	start(com.google.cloud.dataflow.sdk.runners.worker.windmill.Windmill.WorkItem work, org.joda.time.Instant inputDataWatermark, WindmillStateReader stateReader, com.google.cloud.dataflow.sdk.runners.worker.windmill.Windmill.WorkItemCommitRequest.Builder outputBuilder)

Methods inherited from class com.google.cloud.dataflow.sdk.runners.worker.DataflowExecutionContext

withoutSideInputs

Methods inherited from class com.google.cloud.dataflow.sdk.util.BatchModeExecutionContext

getKey, setKey, switchStateKey

Methods inherited from class com.google.cloud.dataflow.sdk.util.BaseExecutionContext

getAllStepContexts, getStepContext, noteOutput, noteSideOutput

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

StreamingModeExecutionContext

public StreamingModeExecutionContext(StateFetcher stateFetcher,
                                     ConcurrentMap<com.google.protobuf.ByteString,UnboundedSource.UnboundedReader<?>> readerCache,
                                     ConcurrentMap<String,String> stateNameMap)

Method Detail

start

public void start(com.google.cloud.dataflow.sdk.runners.worker.windmill.Windmill.WorkItem work,
                  org.joda.time.Instant inputDataWatermark,
                  WindmillStateReader stateReader,
                  com.google.cloud.dataflow.sdk.runners.worker.windmill.Windmill.WorkItemCommitRequest.Builder outputBuilder)

createStepContext

public ExecutionContext.StepContext createStepContext(String stepName,
                                                      String transformName)

Description copied from class: BatchModeExecutionContext

Create a new ExecutionContext.StepContext.

Overrides:

createStepContext in class BatchModeExecutionContext

getSideInputReader

public SideInputReader getSideInputReader(Iterable<? extends com.google.api.services.dataflow.model.SideInputInfo> sideInputInfos)

Description copied from class: DataflowExecutionContext

Returns a SideInputReader for all the side inputs described in the given SideInputInfo descriptors. By default, throws UnsupportedOperationException. Individual workers should override this behavior.

Specified by:

getSideInputReader in class DataflowExecutionContext

getSideInputReaderForViews

public SideInputReader getSideInputReaderForViews(Iterable<? extends PCollectionView<?>> views)

Description copied from class: DataflowExecutionContext

Returns a SideInputReader for all the provided views, where the execution context itself knows how to read data for the view. By default, throws UnsupportedOperationException. Particular workers should override this behavior.

Specified by:

getSideInputReaderForViews in class DataflowExecutionContext

getSideInputNotifications

public Iterable<com.google.cloud.dataflow.sdk.runners.worker.windmill.Windmill.GlobalDataId> getSideInputNotifications()

getSerializedKey

public com.google.protobuf.ByteString getSerializedKey()

getWorkToken

public long getWorkToken()

getWork

public com.google.cloud.dataflow.sdk.runners.worker.windmill.Windmill.WorkItem getWork()

getOutputBuilder

public com.google.cloud.dataflow.sdk.runners.worker.windmill.Windmill.WorkItemCommitRequest.Builder getOutputBuilder()

getCachedReader

public UnboundedSource.UnboundedReader<?> getCachedReader()

setActiveReader

public void setActiveReader(UnboundedSource.UnboundedReader<?> reader)

getReaderCheckpoint

public UnboundedSource.CheckpointMark getReaderCheckpoint(Coder<? extends UnboundedSource.CheckpointMark> coder)

flushState

public Map<Long,Runnable> flushState()

getReadyCommitCallbackIds

public List<Long> getReadyCommitCallbackIds()