Package com.google.cloud.dataflow.sdk.util

Defines utilities used by the Dataflow SDK.

See: Description

Interface Summary

Interface	Description
ActiveWindowSet<W extends BoundedWindow>	Tracks the windows that are active.
ActiveWindowSet.MergeCallback<W extends BoundedWindow>	Callback for ActiveWindowSet.mergeIfAppropriate(W, com.google.cloud.dataflow.sdk.util.ActiveWindowSet.MergeCallback<W>).
CredentialFactory	Construct an oauth credential to be used by the SDK and the SDK workers.
DoFnRunner.OutputManager	Information about how to create output receivers and output to them.
ExecutionContext	Context for the current execution.
ExecutionContext.StepContext	Per-step, per-key context used for retrieving state.
IOChannelFactory	Defines a factory for working with read and write channels.
MonitoringUtil.JobMessagesHandler	An interface that can be used for defining callbacks to receive a list of JobMessages containing monitoring information.
MutationDetector	An object for detecting illegal mutations.
PathValidator	Interface for controlling validation of paths.
ReduceFn.MergingStateContext	Interface for interacting with persistent state within ReduceFn.onMerge(com.google.cloud.dataflow.sdk.util.ReduceFn<K, InputT, OutputT, W>.OnMergeContext).
ReduceFn.StateContext	Interface for interacting with persistent state.
ReduceFn.Timers	Interface for interacting with time.
SideInputReader	The interface to objects that provide side inputs.
SizedSideInputReader	Extension to SideInputReader that can approximate the size of the side input.
Stager	Interface for staging files needed for running a Dataflow pipeline.
TimerInternals	Encapsulate interaction with time within the execution environment.
WindowingInternals<InputT,OutputT>	Interface that may be required by some (internal) DoFns to implement windowing.

Class Summary

Class	Description
ApiSurface	Represents the API surface of a package prefix.
AppEngineEnvironment	Stores whether we are running within AppEngine or not.
AppliedCombineFn<K,InputT,AccumT,OutputT>	A Combine.KeyedCombineFn with a fixed accumulator coder.
AssignWindowsDoFn<T,W extends BoundedWindow>	DoFn that tags elements of a PCollection with windows, according to the provided WindowFn.
AttemptAndTimeBoundedExponentialBackOff	Extension of AttemptBoundedExponentialBackOff that bounds the total time that the backoff is happening as well as the amount of retries.
AttemptBoundedExponentialBackOff	Implementation of BackOff that increases the back off period for each retry attempt using a randomization function that grows exponentially.
Base64Utils	Utilities related to Base64 encoding.
BaseExecutionContext	Base class for implementations of ExecutionContext.
BaseExecutionContext.StepContext	Base class for implementations of ExecutionContext.StepContext.
BatchModeExecutionContext	ExecutionContext for use in batch mode.
BatchTimerInternals	TimerInternals that uses priority queues to manage the timers that are ready to fire.
BigQueryTableInserter	Inserts rows into BigQuery.
BigQueryTableRowIterator	Iterates over all rows in a table.
BoundedQueueExecutor	Executor that blocks on execute() if its queue is full.
CloudCounterUtils	Utilities for working with CloudCounters.
CloudMetricUtils	Utilities for working with Dataflow API Metrics.
CloudObject	A representation of an arbitrary Java object to be instantiated by Dataflow workers.
CloudSourceUtils	Utilities for working with Source Dataflow API definitions and Reader objects.
CoderUtils	Utilities for working with Coders.
CounterAggregator<InputT,AccumT,OutputT>	An implementation of the Aggregator interface that uses a Counter as the underlying representation.
Credentials	Provides support for loading credentials.
DataflowPathValidator	GCP implementation of PathValidator.
DataflowReleaseInfo	Utilities for working with the Dataflow distribution.
DirectModeExecutionContext	ExecutionContext for use in direct mode.
DirectSideInputReader	Basic side input reader wrapping a PTuple of side input iterables.
DoFnInfo<InputT,OutputT>	Wrapper class holding the necessary information to serialize a DoFn.
DoFnRunner<InputT,OutputT>	Runs a DoFn by constructing the appropriate contexts and passing them in.
DoFnRunner.ListOutputManager	An implementation of DoFnRunner.OutputManager using simple lists, for testing and in-memory contexts such as the DirectPipelineRunner.
ExecutableTrigger<W extends BoundedWindow>	A wrapper around a trigger used during execution.
ExposedByteArrayInputStream	ByteArrayInputStream that allows accessing the entire internal buffer without copying.
ExposedByteArrayOutputStream	ByteArrayOutputStream special cased to treat writes of a single byte-array specially.
FileIOChannelFactory	Implements IOChannelFactory for local files.
GcpCredentialFactory	Construct an oauth credential to be used by the SDK and the SDK workers.
GcsIOChannelFactory	Implements IOChannelFactory for GCS.
GcsStager	Utility class for staging files to GCS.
GcsUtil	Provides operations on GCS.
GcsUtil.GcsUtilFactory	This is a DefaultValueFactory able to create a GcsUtil using any transport flags specified on the PipelineOptions.
GroupAlsoByWindowsDoFn<K,InputT,OutputT,W extends BoundedWindow>	DoFn that merges windows and groups elements in those windows, optionally combining values.
InstanceBuilder<T>	Utility for creating objects dynamically.
IntervalBoundedExponentialBackOff	Implementation of BackOff that increases the back off period for each retry attempt using a randomization function that grows exponentially.
IOChannelUtils	Provides utilities for creating read and write channels.
MergingActiveWindowSet<W extends BoundedWindow>	Implementation of ActiveWindowSet used with WindowFns that support merging.
MimeTypes	Constants representing various mime types.
MonitoringUtil	A helper class for monitoring jobs submitted to the service.
MonitoringUtil.PrintHandler	A handler that prints monitoring messages to a stream.
MonitoringUtil.TimeStampComparator	Comparator for sorting rows in increasing order based on timestamp.
MutationDetectors	Static methods for creating and working with MutationDetector.
NonEmptyPanes<W extends BoundedWindow>	Tracks which windows have non-empty panes.
NonMergingActiveWindowSet<W extends BoundedWindow>	Implementation of ActiveWindowSet used with WindowFns that don't support merging.
NoopCredentialFactory	Construct an oauth credential to be used by the SDK and the SDK workers.
NoopPathValidator	Noop implementation of PathValidator.
NullSideInputReader	A SideInputReader representing a well-defined set of views, but not storing any values for them.
OutputReference	A representation used by Steps to reference the output of other Steps.
PackageUtil	Helper routines for packages.
PaneInfoTracker	Encapsulates the logic for tracking the current PaneInfo and producing new PaneInfo for a trigger firing.
PCollectionViews	Implementations of PCollectionView shared across the SDK.
PCollectionViewWindow<T>	A pair of a PCollectionView and a BoundedWindow, which can be thought of as window "of" the view.
PropertyNames	Constant property names used by the SDK in CloudWorkflow specifications.
PTuple	A PTuple is an immutable tuple of heterogeneously-typed values, "keyed" by TupleTags.
ReaderUtils	Utilities for working with Reader objects.
ReduceFn<K,InputT,OutputT,W extends BoundedWindow>	Specification for processing to happen after elements have been grouped by key.
ReduceFnRunner<K,InputT,OutputT,W extends BoundedWindow>	Manages the execution of a ReduceFn after a GroupByKey.GroupByKeyOnly has partitioned the PCollection by key.
ReifyTimestampAndWindowsDoFn<K,V>	DoFn that makes timestamps and window assignments explicit in the value part of each key/value pair.
Reshuffle<K,V>	A PTransform to reshuffle the elements based on their key.
RetryHttpRequestInitializer	Implements a request initializer that adds retry handlers to all HttpRequests.
SerializableUtils	Utilities for working with Serializables.
Serializer	Utility for converting objects between Java and Cloud representations.
ShardingWritableByteChannel	Implements a WritableByteChannel that may contain multiple output shards.
Sized<T>	A T with an accompanying size estimate.
SizedSideInputReader.Defaults	Abstract class providing default implementations for methods of SizedSideInputReader.
StateFetcher	Class responsible for fetching state from the windmill server.
StreamingGroupAlsoByWindowsDoFn<K,InputT,OutputT,W extends BoundedWindow>	DoFn that merges windows and groups elements in those windows.
StreamingModeExecutionContext	ExecutionContext for use in streaming mode.
StreamingModeExecutionContext.StreamingModeSideInputReader	A SideInputReader that fetches side inputs from the streaming worker's cache.
StreamingPCollectionViewWriterFn<T>	Utility DoFn for writing streaming PCollectionViews.
StreamingSideInputDoFnRunner<InputT,OutputT,W extends BoundedWindow>	Runs a DoFn by constructing the appropriate contexts and passing them in.
StreamUtils	Utility functions for stream operations.
StringUtils	Utilities for working with JSON and other human-readable string formats.
Structs	A collection of static methods for manipulating datastructure representations transferred via the Dataflow API.
TestCredential	Fake credential, for use in testing.
TimerInternals.TimerData	Data about a timer as represented within TimerInternals.
TimerOrElement<ElemT>	Class representing either a timer, or arbitrary element.
TimerOrElement.TimerOrElementCoder<T>	Coder that forwards ByteSizeObserver calls to an underlying element coder.
TimeUtil	A helper class for converting between Dataflow API and SDK time representations.
Transport	Helpers for cloud communication.
TriggerRunner<W extends BoundedWindow>	Executes a trigger within the context provided by ReduceFnRunner.
TriggerRunner.Result	Result of trigger execution.
Values	A collection of static methods for manipulating value representations transfered via the Dataflow API.
ValueWithRecordId<ValueT>	Immutable struct containing a value as well as a unique id identifying the value.
ValueWithRecordId.ValueWithRecordIdCoder<ValueT>	A Coder for ValueWithRecordId, using a wrapped value Coder.
VarInt	Variable-length encoding for integers.
WatermarkHold<W extends BoundedWindow>	Implements the logic needed to hold the output watermark back to emit values with specific timestamps.
WindowedValue<V>	An immutable triple of value, timestamp, and windows.
WindowedValue.FullWindowedValueCoder<T>	Coder for WindowedValue.
WindowedValue.ValueOnlyWindowedValueCoder<T>	Coder for WindowedValue.
WindowedValue.WindowedValueCoder<T>	Abstract class for WindowedValue coder.
WindowingStrategy<T,W extends BoundedWindow>	A WindowingStrategy describes the windowing behavior for a specific collection of values.

Enum Summary

Enum	Description
StateFetcher.SideInputState	Indicates the caller's knowledge of whether a particular side input has been computed.
TimeDomain	TimeDomain specifies whether an operation is based on timestamps of elements or current "real-world" time as reported while processing.
WindowingStrategy.AccumulationMode	The accumulation modes that can be used with windowing.

Exception Summary

Exception	Description
IllegalMutationException	Thrown when a value appears to have been mutated, but that mutation is forbidden.
UserCodeException	An exception that was thrown in user-code.

Annotation Types Summary

Annotation Type	Description
SystemDoFnInternal	Annotation to mark DoFns as an internal component of the Dataflow SDK.

Package com.google.cloud.dataflow.sdk.util Description

Defines utilities used by the Dataflow SDK.