DataStream

Instance Constructors

new DataStream(stream: datastream.DataStream[T])

Value Members

final def !=(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def ##(): Int

Definition Classes
AnyRef → Any
final def ==(arg0: Any): Boolean

Definition Classes
AnyRef → Any
def addSink(fun: (T) ⇒ Unit): DataStreamSink[T]

Adds the given sink to this DataStream.
Adds the given sink to this DataStream. Only streams with sinks added will be executed once the StreamExecutionEnvironment.execute(...) method is called.
def addSink(sinkFunction: SinkFunction[T]): DataStreamSink[T]

Adds the given sink to this DataStream.
Adds the given sink to this DataStream. Only streams with sinks added will be executed once the StreamExecutionEnvironment.execute(...) method is called.
final def asInstanceOf[T0]: T0

Definition Classes
Any
def assignAscendingTimestamps(extractor: (T) ⇒ Long): DataStream[T]

Assigns timestamps to the elements in the data stream and periodically creates watermarks to signal event time progress.
Assigns timestamps to the elements in the data stream and periodically creates watermarks to signal event time progress.
This method is a shortcut for data streams where the element timestamp are known to be monotonously ascending within each parallel stream. In that case, the system can generate watermarks automatically and perfectly by tracking the ascending timestamps.
For cases where the timestamps are not monotonously increasing, use the more general methods assignTimestampsAndWatermarks(AssignerWithPeriodicWatermarks) and assignTimestampsAndWatermarks(AssignerWithPunctuatedWatermarks).

Annotations
@PublicEvolving()
def assignTimestampsAndWatermarks(assigner: AssignerWithPunctuatedWatermarks[T]): DataStream[T]

Assigns timestamps to the elements in the data stream and periodically creates watermarks to signal event time progress.
Assigns timestamps to the elements in the data stream and periodically creates watermarks to signal event time progress.
This method creates watermarks based purely on stream elements. For each element that is handled via long), the AssignerWithPunctuatedWatermarks#checkAndGetNextWatermark() method is called, and a new watermark is emitted, if the returned watermark value is larger than the previous watermark.
This method is useful when the data stream embeds watermark elements, or certain elements carry a marker that can be used to determine the current event time watermark. This operation gives the programmer full control over the watermark generation. Users should be aware that too aggressive watermark generation (i.e., generating hundreds of watermarks every second) can cost some performance.
For cases where watermarks should be created in a regular fashion, for example every x milliseconds, use the AssignerWithPeriodicWatermarks.

Annotations
@PublicEvolving()
See also
#assignTimestampsAndWatermarks(AssignerWithPeriodicWatermarks)
AssignerWithPeriodicWatermarks
AssignerWithPunctuatedWatermarks
def assignTimestampsAndWatermarks(assigner: AssignerWithPeriodicWatermarks[T]): DataStream[T]

Assigns timestamps to the elements in the data stream and periodically creates watermarks to signal event time progress.
Assigns timestamps to the elements in the data stream and periodically creates watermarks to signal event time progress.
This method creates watermarks periodically (for example every second), based on the watermarks indicated by the given watermark generator. Even when no new elements in the stream arrive, the given watermark generator will be periodically checked for new watermarks. The interval in which watermarks are generated is defined in org.apache.flink.api.common.ExecutionConfig#setAutoWatermarkInterval(long).
Use this method for the common cases, where some characteristic over all elements should generate the watermarks, or where watermarks are simply trailing behind the wall clock time by a certain amount.
For the second case and when the watermarks are required to lag behind the maximum timestamp seen so far in the elements of the stream by a fixed amount of time, and this amount is known in advance, use the BoundedOutOfOrdernessTimestampExtractor.
For cases where watermarks should be created in an irregular fashion, for example based on certain markers that some element carry, use the AssignerWithPunctuatedWatermarks.

Annotations
@PublicEvolving()
See also
#assignTimestampsAndWatermarks(AssignerWithPunctuatedWatermarks)
AssignerWithPunctuatedWatermarks
AssignerWithPeriodicWatermarks
def broadcast(broadcastStateDescriptors: MapStateDescriptor[_, _]*): BroadcastStream[T]

Sets the partitioning of the DataStream so that the output elements are broadcasted to every parallel instance of the next operation.
Sets the partitioning of the DataStream so that the output elements are broadcasted to every parallel instance of the next operation. In addition, it implicitly creates as many broadcast states as the specified descriptors which can be used to store the element of the stream.
broadcastStateDescriptors
the descriptors of the broadcast states to create.
returns
A BroadcastStream which can be used in the DataStream.connect(BroadcastStream) to create a BroadcastConnectedStream for further processing of the elements.

Annotations
@PublicEvolving()
def broadcast: DataStream[T]

Sets the partitioning of the DataStream so that the output tuples are broad casted to every parallel instance of the next component.
def clone(): AnyRef

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( ... )
def coGroup[T2](otherStream: DataStream[T2]): CoGroupedStreams[T, T2]

Creates a co-group operation.
Creates a co-group operation. See CoGroupedStreams for an example of how the keys and window can be specified.
def connect[R](broadcastStream: BroadcastStream[R]): BroadcastConnectedStream[T, R]

Creates a new BroadcastConnectedStream by connecting the current DataStream or KeyedStream with a BroadcastStream.
Creates a new BroadcastConnectedStream by connecting the current DataStream or KeyedStream with a BroadcastStream.
The latter can be created using the broadcast(MapStateDescriptor[]) method.
The resulting stream can be further processed using the broadcastConnectedStream.process(myFunction) method, where myFunction can be either a org.apache.flink.streaming.api.functions.co.KeyedBroadcastProcessFunction or a org.apache.flink.streaming.api.functions.co.BroadcastProcessFunction depending on the current stream being a KeyedStream or not.
broadcastStream
The broadcast stream with the broadcast state to be connected with this stream.
returns
The BroadcastConnectedStream.

Annotations
@PublicEvolving()
def connect[T2](dataStream: DataStream[T2]): ConnectedStreams[T, T2]

Creates a new ConnectedStreams by connecting DataStream outputs of different type with each other.
Creates a new ConnectedStreams by connecting DataStream outputs of different type with each other. The DataStreams connected using this operators can be used with CoFunctions.
def countWindowAll(size: Long): AllWindowedStream[T, GlobalWindow]

Windows this DataStream into tumbling count windows.
Windows this DataStream into tumbling count windows.
Note: This operation can be inherently non-parallel since all elements have to pass through the same operator instance. (Only for special cases, such as aligned time windows is it possible to perform this operation in parallel).
size
The size of the windows in number of elements.
def countWindowAll(size: Long, slide: Long): AllWindowedStream[T, GlobalWindow]

Windows this DataStream into sliding count windows.
Windows this DataStream into sliding count windows.
Note: This operation can be inherently non-parallel since all elements have to pass through the same operator instance. (Only for special cases, such as aligned time windows is it possible to perform this operation in parallel).
size
The size of the windows in number of elements.
slide
The slide interval in number of elements.
def dataType: TypeInformation[T]

Returns the TypeInformation for the elements of this DataStream.
def disableChaining(): DataStream[T]

Turns off chaining for this operator so thread co-location will not be used as an optimization.
Turns off chaining for this operator so thread co-location will not be used as an optimization. Chaining can be turned off for the whole job by StreamExecutionEnvironment.disableOperatorChaining() however it is not advised for performance considerations.

Annotations
@PublicEvolving()
final def eq(arg0: AnyRef): Boolean

Definition Classes
AnyRef
def equals(arg0: Any): Boolean

Definition Classes
AnyRef → Any
def executionConfig: ExecutionConfig

Returns the execution config.
def executionEnvironment: StreamExecutionEnvironment

Returns the StreamExecutionEnvironment associated with this data stream
def filter(fun: (T) ⇒ Boolean): DataStream[T]

Creates a new DataStream that contains only the elements satisfying the given filter predicate.
def filter(filter: FilterFunction[T]): DataStream[T]

Creates a new DataStream that contains only the elements satisfying the given filter predicate.
def finalize(): Unit

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( classOf[java.lang.Throwable] )
def flatMap[R](fun: (T) ⇒ TraversableOnce[R])(implicit arg0: TypeInformation[R]): DataStream[R]

Creates a new DataStream by applying the given function to every element and flattening the results.
def flatMap[R](fun: (T, Collector[R]) ⇒ Unit)(implicit arg0: TypeInformation[R]): DataStream[R]

Creates a new DataStream by applying the given function to every element and flattening the results.
def flatMap[R](flatMapper: FlatMapFunction[T, R])(implicit arg0: TypeInformation[R]): DataStream[R]

Creates a new DataStream by applying the given function to every element and flattening the results.
def forward: DataStream[T]

Sets the partitioning of the DataStream so that the output tuples are forwarded to the local subtask of the next component (whenever possible).
final def getClass(): Class[_]

Definition Classes
AnyRef → Any
def getSideOutput[X](tag: OutputTag[X])(implicit arg0: TypeInformation[X]): DataStream[X]

Annotations
@PublicEvolving()
def global: DataStream[T]

Sets the partitioning of the DataStream so that the output values all go to the first instance of the next processing operator.
Sets the partitioning of the DataStream so that the output values all go to the first instance of the next processing operator. Use this setting with care since it might cause a serious performance bottleneck in the application.

Annotations
@PublicEvolving()
def hashCode(): Int

Definition Classes
AnyRef → Any
final def isInstanceOf[T0]: Boolean

Definition Classes
Any
def iterate[R, F](stepFunction: (ConnectedStreams[T, F]) ⇒ (DataStream[F], DataStream[R]), maxWaitTimeMillis: Long)(implicit arg0: TypeInformation[F]): DataStream[R]

Initiates an iterative part of the program that creates a loop by feeding back data streams.
Initiates an iterative part of the program that creates a loop by feeding back data streams. To create a streaming iteration the user needs to define a transformation that creates two DataStreams. The first one is the output that will be fed back to the start of the iteration and the second is the output stream of the iterative part.
The input stream of the iterate operator and the feedback stream will be treated as a ConnectedStreams where the input is connected with the feedback stream.
This allows the user to distinguish standard input from feedback inputs.
stepfunction: initialStream => (feedback, output)
The user must set the max waiting time for the iteration head. If no data received in the set time the stream terminates. If this parameter is set to 0 then the iteration sources will indefinitely, so the job must be killed to stop.

Annotations
@PublicEvolving()
def iterate[R](stepFunction: (DataStream[T]) ⇒ (DataStream[T], DataStream[R]), maxWaitTimeMillis: Long = 0): DataStream[R]

Initiates an iterative part of the program that creates a loop by feeding back data streams.
Initiates an iterative part of the program that creates a loop by feeding back data streams. To create a streaming iteration the user needs to define a transformation that creates two DataStreams. The first one is the output that will be fed back to the start of the iteration and the second is the output stream of the iterative part.
stepfunction: initialStream => (feedback, output)
A common pattern is to use output splitting to create feedback and output DataStream. Please refer to the split method of the DataStream
By default a DataStream with iteration will never terminate, but the user can use the maxWaitTime parameter to set a max waiting time for the iteration head. If no data received in the set time the stream terminates.
Parallelism of the feedback stream must match the parallelism of the original stream. Please refer to the setParallelism method for parallelism modification

Annotations
@PublicEvolving()
def javaStream: datastream.DataStream[T]

Gets the underlying java DataStream object.
def join[T2](otherStream: DataStream[T2]): JoinedStreams[T, T2]

Creates a join operation.
Creates a join operation. See JoinedStreams for an example of how the keys and window can be specified.
def keyBy[K](fun: (T) ⇒ K)(implicit arg0: TypeInformation[K]): KeyedStream[T, K]

Groups the elements of a DataStream by the given K key to be used with grouped operators like grouped reduce or grouped aggregations.
def keyBy(firstField: String, otherFields: String*): KeyedStream[T, Tuple]

Groups the elements of a DataStream by the given field expressions to be used with grouped operators like grouped reduce or grouped aggregations.
def keyBy(fields: Int*): KeyedStream[T, Tuple]

Groups the elements of a DataStream by the given key positions (for tuple/array types) to be used with grouped operators like grouped reduce or grouped aggregations.
def map[R](mapper: MapFunction[T, R])(implicit arg0: TypeInformation[R]): DataStream[R]

Creates a new DataStream by applying the given function to every element of this DataStream.
def map[R](fun: (T) ⇒ R)(implicit arg0: TypeInformation[R]): DataStream[R]

Creates a new DataStream by applying the given function to every element of this DataStream.
def minResources: ResourceSpec

Returns the minimum resources of this operation.
Returns the minimum resources of this operation.

Annotations
@PublicEvolving()
def name(name: String): DataStream[T]

Sets the name of the current data stream.
Sets the name of the current data stream. This name is used by the visualization and logging during runtime.
returns
The named operator
def name: String

Gets the name of the current data stream.
Gets the name of the current data stream. This name is used by the visualization and logging during runtime.
returns
Name of the stream.
final def ne(arg0: AnyRef): Boolean

Definition Classes
AnyRef
final def notify(): Unit

Definition Classes
AnyRef
final def notifyAll(): Unit

Definition Classes
AnyRef
def parallelism: Int

Returns the parallelism of this operation.
def partitionCustom[K](partitioner: Partitioner[K], fun: (T) ⇒ K)(implicit arg0: TypeInformation[K]): DataStream[T]

Partitions a DataStream on the key returned by the selector, using a custom partitioner.
Partitions a DataStream on the key returned by the selector, using a custom partitioner. This method takes the key selector to get the key to partition on, and a partitioner that accepts the key type.
Note: This method works only on single field keys, i.e. the selector cannot return tuples of fields.
def partitionCustom[K](partitioner: Partitioner[K], field: String)(implicit arg0: TypeInformation[K]): DataStream[T]

Partitions a POJO DataStream on the specified key fields using a custom partitioner.
Partitions a POJO DataStream on the specified key fields using a custom partitioner. This method takes the key expression to partition on, and a partitioner that accepts the key type.
Note: This method works only on single field keys.
def partitionCustom[K](partitioner: Partitioner[K], field: Int)(implicit arg0: TypeInformation[K]): DataStream[T]

Partitions a tuple DataStream on the specified key fields using a custom partitioner.
Partitions a tuple DataStream on the specified key fields using a custom partitioner. This method takes the key position to partition on, and a partitioner that accepts the key type.
Note: This method works only on single field keys.
def preferredResources: ResourceSpec

Returns the preferred resources of this operation.
Returns the preferred resources of this operation.

Annotations
@PublicEvolving()
def print(): DataStreamSink[T]

Writes a DataStream to the standard output stream (stdout).
Writes a DataStream to the standard output stream (stdout). For each element of the DataStream the result of .toString is written.

Annotations
@PublicEvolving()
def printToErr(): DataStreamSink[T]

Writes a DataStream to the standard output stream (stderr).
Writes a DataStream to the standard output stream (stderr).
For each element of the DataStream the result of AnyRef.toString() is written.
returns
The closed DataStream.

Annotations
@PublicEvolving()
def process[R](processFunction: ProcessFunction[T, R])(implicit arg0: TypeInformation[R]): DataStream[R]

Applies the given ProcessFunction on the input stream, thereby creating a transformed output stream.
Applies the given ProcessFunction on the input stream, thereby creating a transformed output stream.
The function will be called for every element in the stream and can produce zero or more output.
processFunction
The ProcessFunction that is called for each element in the stream.

Annotations
@PublicEvolving()
def rebalance: DataStream[T]

Sets the partitioning of the DataStream so that the output tuples are distributed evenly to the next component.
def rescale: DataStream[T]

Sets the partitioning of the DataStream so that the output tuples are distributed evenly to a subset of instances of the downstream operation.
Sets the partitioning of the DataStream so that the output tuples are distributed evenly to a subset of instances of the downstream operation.
The subset of downstream operations to which the upstream operation sends elements depends on the degree of parallelism of both the upstream and downstream operation. For example, if the upstream operation has parallelism 2 and the downstream operation has parallelism 4, then one upstream operation would distribute elements to two downstream operations while the other upstream operation would distribute to the other two downstream operations. If, on the other hand, the downstream operation has parallelism 2 while the upstream operation has parallelism 4 then two upstream operations will distribute to one downstream operation while the other two upstream operations will distribute to the other downstream operations.
In cases where the different parallelisms are not multiples of each other one or several downstream operations will have a differing number of inputs from upstream operations.

Annotations
@PublicEvolving()
def setBufferTimeout(timeoutMillis: Long): DataStream[T]

Sets the maximum time frequency (ms) for the flushing of the output buffer.
Sets the maximum time frequency (ms) for the flushing of the output buffer. By default the output buffers flush only when they are full.
timeoutMillis
The maximum time between two output flushes.
returns
The operator with buffer timeout set.
def setConfigItem(key: String, value: String): DataStream[T]

Sets the value of the given option for the operator.
Sets the value of the given option for the operator.
key
The name of the option to be updated.
value
The value of the option to be updated.
def setConfigItem(key: ConfigOption[String], value: String): DataStream[T]

Sets the value of the given option for the operator.
Sets the value of the given option for the operator.
key
The option to be updated.
value
The value of the option to be updated.
def setMaxParallelism(maxParallelism: Int): DataStream[T]
def setParallelism(parallelism: Int): DataStream[T]

Sets the parallelism of this operation.
Sets the parallelism of this operation. This must be at least 1.
def setResourceConstraints(resourceConstraints: ResourceConstraints): DataStream[T]

Annotations
@PublicEvolving()
def setResources(resources: ResourceSpec): DataStream[T]

Sets the resource of this operation.
Sets the resource of this operation.

Annotations
@PublicEvolving()
def setResources(minResources: ResourceSpec, preferredResources: ResourceSpec): DataStream[T]

Sets the minimum and preferred resources of this operation.
Sets the minimum and preferred resources of this operation.

Annotations
@PublicEvolving()
def setUidHash(hash: String): DataStream[T]

Sets an user provided hash for this operator.
Sets an user provided hash for this operator. This will be used AS IS the create the JobVertexID.
The user provided hash is an alternative to the generated hashes, that is considered when identifying an operator through the default hash mechanics fails (e.g. because of changes between Flink versions).
Important: this should be used as a workaround or for trouble shooting. The provided hash needs to be unique per transformation and job. Otherwise, job submission will fail. Furthermore, you cannot assign user-specified hash to intermediate nodes in an operator chain and trying so will let your job fail.
hash
the user provided hash for this operator.
returns
The operator with the user provided hash.

Annotations
@PublicEvolving()
def shuffle: DataStream[T]

Sets the partitioning of the DataStream so that the output tuples are shuffled to the next component.
Sets the partitioning of the DataStream so that the output tuples are shuffled to the next component.

Annotations
@PublicEvolving()
def slotSharingGroup(slotSharingGroup: String): DataStream[T]

Sets the slot sharing group of this operation.
Sets the slot sharing group of this operation. Parallel instances of operations that are in the same slot sharing group will be co-located in the same TaskManager slot, if possible.
Operations inherit the slot sharing group of input operations if all input operations are in the same slot sharing group and no slot sharing group was explicitly specified.
Initially an operation is in the default slot sharing group. An operation can be put into the default group explicitly by setting the slot sharing group to "default".
slotSharingGroup
The slot sharing group name.

Annotations
@PublicEvolving()
def split(fun: (T) ⇒ TraversableOnce[String]): SplitStream[T]

Creates a new SplitStream that contains only the elements satisfying the given output selector predicate.
def split(selector: OutputSelector[T]): SplitStream[T]

Operator used for directing tuples to specific named outputs using an OutputSelector.
Operator used for directing tuples to specific named outputs using an OutputSelector. Calling this method on an operator creates a new SplitStream.
def startNewChain(): DataStream[T]

Starts a new task chain beginning at this operator.
Starts a new task chain beginning at this operator. This operator will not be chained (thread co-located for increased performance) to any previous tasks even if possible.

Annotations
@PublicEvolving()
final def synchronized[T0](arg0: ⇒ T0): T0

Definition Classes
AnyRef
def timeWindowAll(size: Time, slide: Time): AllWindowedStream[T, TimeWindow]

Windows this DataStream into sliding time windows.
Windows this DataStream into sliding time windows.
This is a shortcut for either .window(SlidingEventTimeWindows.of(size, slide)) or .window(SlidingProcessingTimeWindows.of(size, slide)) depending on the time characteristic set using StreamExecutionEnvironment.setStreamTimeCharacteristic.
Note: This operation can be inherently non-parallel since all elements have to pass through the same operator instance. (Only for special cases, such as aligned time windows is it possible to perform this operation in parallel).
size
The size of the window.
def timeWindowAll(size: Time): AllWindowedStream[T, TimeWindow]

Windows this DataStream into tumbling time windows.
Windows this DataStream into tumbling time windows.
This is a shortcut for either .window(TumblingEventTimeWindows.of(size)) or .window(TumblingProcessingTimeWindows.of(size)) depending on the time characteristic set using StreamExecutionEnvironment.setStreamTimeCharacteristic.
Note: This operation can be inherently non-parallel since all elements have to pass through the same operator instance. (Only for special cases, such as aligned time windows is it possible to perform this operation in parallel).
size
The size of the window.
def toString(): String

Definition Classes
AnyRef → Any
def transform[R](operatorName: String, operator: OneInputStreamOperator[T, R])(implicit arg0: TypeInformation[R]): DataStream[R]

Transforms the DataStream by using a custom OneInputStreamOperator.
Transforms the DataStream by using a custom OneInputStreamOperator.
R
the type of elements emitted by the operator
operatorName
name of the operator, for logging purposes
operator
the object containing the transformation logic

Annotations
@PublicEvolving()
def uid(uid: String): DataStream[T]

Sets an ID for this operator.
Sets an ID for this operator.
The specified ID is used to assign the same operator ID across job submissions (for example when starting a job from a savepoint).
Important: this ID needs to be unique per transformation and job. Otherwise, job submission will fail.
uid
The unique user-specified ID of this transformation.
returns
The operator with the specified ID.

Annotations
@PublicEvolving()
def union(dataStreams: DataStream[T]*): DataStream[T]

Creates a new DataStream by merging DataStream outputs of the same type with each other.
Creates a new DataStream by merging DataStream outputs of the same type with each other. The DataStreams merged using this operator will be transformed simultaneously.
final def wait(): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long, arg1: Int): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
def windowAll[W <: Window](assigner: WindowAssigner[_ >: T, W]): AllWindowedStream[T, W]

Windows this data stream to a AllWindowedStream, which evaluates windows over a key grouped stream.
Windows this data stream to a AllWindowedStream, which evaluates windows over a key grouped stream. Elements are put into windows by a WindowAssigner. The grouping of elements is done both by key and by window.
A org.apache.flink.streaming.api.windowing.triggers.Trigger can be defined to specify when windows are evaluated. However, WindowAssigner have a default Trigger that is used if a Trigger is not specified.
Note: This operation can be inherently non-parallel since all elements have to pass through the same operator instance. (Only for special cases, such as aligned time windows is it possible to perform this operation in parallel).
assigner
The WindowAssigner that assigns elements to windows.
returns
The trigger windows data stream.

Annotations
@PublicEvolving()
def writeAsCsv(path: String, writeMode: WriteMode, rowDelimiter: String, fieldDelimiter: String): DataStreamSink[T]

Writes the DataStream in CSV format to the file specified by the path parameter.
Writes the DataStream in CSV format to the file specified by the path parameter. The writing is performed periodically every millis milliseconds.
path
Path to the location of the CSV file
writeMode
Controls whether an existing file is overwritten or not
rowDelimiter
Delimiter for consecutive rows
fieldDelimiter
Delimiter for consecutive fields
returns
The closed DataStream

Annotations
@PublicEvolving()
def writeAsCsv(path: String, writeMode: WriteMode): DataStreamSink[T]

Writes the DataStream in CSV format to the file specified by the path parameter.
Writes the DataStream in CSV format to the file specified by the path parameter. The writing is performed periodically every millis milliseconds.
path
Path to the location of the CSV file
writeMode
Controls whether an existing file is overwritten or not
returns
The closed DataStream

Annotations
@PublicEvolving()
def writeAsCsv(path: String): DataStreamSink[T]

Writes the DataStream in CSV format to the file specified by the path parameter.
Writes the DataStream in CSV format to the file specified by the path parameter. The writing is performed periodically every millis milliseconds.
path
Path to the location of the CSV file
returns
The closed DataStream

Annotations
@PublicEvolving()
def writeAsText(path: String, writeMode: WriteMode): DataStreamSink[T]

Writes a DataStream to the file specified by path in text format.
Writes a DataStream to the file specified by path in text format. For every element of the DataStream the result of .toString is written.
path
The path pointing to the location the text file is written to
writeMode
Controls the behavior for existing files. Options are NO_OVERWRITE and OVERWRITE.
returns
The closed DataStream

Annotations
@PublicEvolving()
def writeAsText(path: String): DataStreamSink[T]

Writes a DataStream to the file specified by path in text format.
Writes a DataStream to the file specified by path in text format. For every element of the DataStream the result of .toString is written.
path
The path pointing to the location the text file is written to
returns
The closed DataStream

Annotations
@PublicEvolving()
def writeToSocket(hostname: String, port: Integer, schema: SerializationSchema[T]): DataStreamSink[T]

Writes the DataStream to a socket as a byte array.
Writes the DataStream to a socket as a byte array. The format of the output is specified by a SerializationSchema.

Annotations
@PublicEvolving()
def writeUsingOutputFormat(format: OutputFormat[T]): DataStreamSink[T]

Writes a DataStream using the given OutputFormat.
Writes a DataStream using the given OutputFormat.

Annotations
@PublicEvolving()

Deprecated Value Members

def assignTimestamps(extractor: TimestampExtractor[T]): DataStream[T]

Extracts a timestamp from an element and assigns it as the internal timestamp of that element.
Extracts a timestamp from an element and assigns it as the internal timestamp of that element. The internal timestamps are, for example, used to to event-time window operations.
If you know that the timestamps are strictly increasing you can use an AscendingTimestampExtractor. Otherwise, you should provide a TimestampExtractor that also implements TimestampExtractor#getCurrentWatermark to keep track of watermarks.

Annotations
@deprecated
Deprecated
See also
org.apache.flink.streaming.api.watermark.Watermark
def getExecutionConfig: ExecutionConfig

Returns the execution config.
Returns the execution config.

Annotations
@deprecated @PublicEvolving()
Deprecated
def getExecutionEnvironment: StreamExecutionEnvironment

Returns the StreamExecutionEnvironment associated with the current DataStream.
Returns the StreamExecutionEnvironment associated with the current DataStream.
returns
associated execution environment

Annotations
@deprecated @PublicEvolving()
Deprecated
def getName: String

Gets the name of the current data stream.
Gets the name of the current data stream. This name is used by the visualization and logging during runtime.
returns
Name of the stream.

Annotations
@deprecated @PublicEvolving()
Deprecated
def getParallelism: Int

Returns the parallelism of this operation.
Returns the parallelism of this operation.

Annotations
@deprecated @PublicEvolving()
Deprecated
def getType(): TypeInformation[T]

Returns the TypeInformation for the elements of this DataStream.
Returns the TypeInformation for the elements of this DataStream.

Annotations
@deprecated @PublicEvolving()
Deprecated

Related Doc: package scala

class DataStream[T] extends AnyRef

Instance Constructors

new DataStream(stream: datastream.DataStream[T])

Value Members

final def !=(arg0: Any): Boolean

final def ##(): Int

final def ==(arg0: Any): Boolean

def addSink(fun: (T) ⇒ Unit): DataStreamSink[T]

def addSink(sinkFunction: SinkFunction[T]): DataStreamSink[T]

final def asInstanceOf[T0]: T0

def assignAscendingTimestamps(extractor: (T) ⇒ Long): DataStream[T]

def assignTimestampsAndWatermarks(assigner: AssignerWithPunctuatedWatermarks[T]): DataStream[T]

def assignTimestampsAndWatermarks(assigner: AssignerWithPeriodicWatermarks[T]): DataStream[T]

def broadcast(broadcastStateDescriptors: MapStateDescriptor[_, _]*): BroadcastStream[T]

def broadcast: DataStream[T]

def clone(): AnyRef

def coGroup[T2](otherStream: DataStream[T2]): CoGroupedStreams[T, T2]

def connect[R](broadcastStream: BroadcastStream[R]): BroadcastConnectedStream[T, R]

def connect[T2](dataStream: DataStream[T2]): ConnectedStreams[T, T2]

def countWindowAll(size: Long): AllWindowedStream[T, GlobalWindow]

def countWindowAll(size: Long, slide: Long): AllWindowedStream[T, GlobalWindow]

def dataType: TypeInformation[T]

def disableChaining(): DataStream[T]

final def eq(arg0: AnyRef): Boolean

def equals(arg0: Any): Boolean

def executionConfig: ExecutionConfig

def executionEnvironment: StreamExecutionEnvironment

def filter(fun: (T) ⇒ Boolean): DataStream[T]

def filter(filter: FilterFunction[T]): DataStream[T]

def finalize(): Unit

def flatMap[R](fun: (T) ⇒ TraversableOnce[R])(implicit arg0: TypeInformation[R]): DataStream[R]

def flatMap[R](fun: (T, Collector[R]) ⇒ Unit)(implicit arg0: TypeInformation[R]): DataStream[R]

def flatMap[R](flatMapper: FlatMapFunction[T, R])(implicit arg0: TypeInformation[R]): DataStream[R]

def forward: DataStream[T]

final def getClass(): Class[_]

def getSideOutput[X](tag: OutputTag[X])(implicit arg0: TypeInformation[X]): DataStream[X]

def global: DataStream[T]

def hashCode(): Int

final def isInstanceOf[T0]: Boolean

def iterate[R, F](stepFunction: (ConnectedStreams[T, F]) ⇒ (DataStream[F], DataStream[R]), maxWaitTimeMillis: Long)(implicit arg0: TypeInformation[F]): DataStream[R]

def iterate[R](stepFunction: (DataStream[T]) ⇒ (DataStream[T], DataStream[R]), maxWaitTimeMillis: Long = 0): DataStream[R]

def javaStream: datastream.DataStream[T]

def join[T2](otherStream: DataStream[T2]): JoinedStreams[T, T2]

def keyBy[K](fun: (T) ⇒ K)(implicit arg0: TypeInformation[K]): KeyedStream[T, K]

def keyBy(firstField: String, otherFields: String*): KeyedStream[T, Tuple]

def keyBy(fields: Int*): KeyedStream[T, Tuple]

def map[R](mapper: MapFunction[T, R])(implicit arg0: TypeInformation[R]): DataStream[R]

def map[R](fun: (T) ⇒ R)(implicit arg0: TypeInformation[R]): DataStream[R]

def minResources: ResourceSpec

def name(name: String): DataStream[T]

def name: String

final def ne(arg0: AnyRef): Boolean

final def notify(): Unit

final def notifyAll(): Unit

def parallelism: Int

def partitionCustom[K](partitioner: Partitioner[K], fun: (T) ⇒ K)(implicit arg0: TypeInformation[K]): DataStream[T]

def partitionCustom[K](partitioner: Partitioner[K], field: String)(implicit arg0: TypeInformation[K]): DataStream[T]

def partitionCustom[K](partitioner: Partitioner[K], field: Int)(implicit arg0: TypeInformation[K]): DataStream[T]

def preferredResources: ResourceSpec

def print(): DataStreamSink[T]

def printToErr(): DataStreamSink[T]

def process[R](processFunction: ProcessFunction[T, R])(implicit arg0: TypeInformation[R]): DataStream[R]

def rebalance: DataStream[T]

def rescale: DataStream[T]

def setBufferTimeout(timeoutMillis: Long): DataStream[T]

def setConfigItem(key: String, value: String): DataStream[T]

def setConfigItem(key: ConfigOption[String], value: String): DataStream[T]

def setMaxParallelism(maxParallelism: Int): DataStream[T]

def setParallelism(parallelism: Int): DataStream[T]

def setResourceConstraints(resourceConstraints: ResourceConstraints): DataStream[T]

def setResources(resources: ResourceSpec): DataStream[T]

def setResources(minResources: ResourceSpec, preferredResources: ResourceSpec): DataStream[T]

def setUidHash(hash: String): DataStream[T]

def shuffle: DataStream[T]

def slotSharingGroup(slotSharingGroup: String): DataStream[T]

def split(fun: (T) ⇒ TraversableOnce[String]): SplitStream[T]

def split(selector: OutputSelector[T]): SplitStream[T]

def startNewChain(): DataStream[T]