org.apache.flink.streaming.python.api.datastream

Class PythonDataStream<D extends org.apache.flink.streaming.api.datastream.DataStream<org.python.core.PyObject>>

java.lang.Object

org.apache.flink.streaming.python.api.datastream.PythonDataStream<D>

Direct Known Subclasses:

PythonKeyedStream, PythonSingleOutputStreamOperator, PythonSplitStream

@PublicEvolving
public class PythonDataStream<D extends org.apache.flink.streaming.api.datastream.DataStream<org.python.core.PyObject>>
extends Object

A PythonDataStream is a thin wrapper layer over DataStream, which represents a stream of elements of the same type. A PythonDataStream can be transformed into another PythonDataStream by applying various transformation functions, such as

• map(org.apache.flink.api.common.functions.MapFunction<org.python.core.PyObject, org.python.core.PyObject>)
• split(org.apache.flink.streaming.api.collector.selector.OutputSelector<org.python.core.PyObject>)

A thin wrapper layer means that the functionality itself is performed by the DataStream, however instead of working directly with the streaming data sets, this layer handles Python wrappers (e.g. PythonDataStream) to comply with the Python standard coding styles.

Field Summary

Fields

Modifier and Type	Field and Description
protected D	stream

Constructor Summary

Constructors

Constructor and Description
PythonDataStream(D stream)

Method Summary

Modifier and Type	Method and Description
void	add_sink(org.apache.flink.streaming.api.functions.sink.SinkFunction<org.python.core.PyObject> sink_func) A thin wrapper layer over DataStream.addSink(SinkFunction).
PythonSingleOutputStreamOperator	filter(org.apache.flink.api.common.functions.FilterFunction<org.python.core.PyObject> filter) A thin wrapper layer over DataStream.filter(FilterFunction).
PythonDataStream<org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator<org.python.core.PyObject>>	flat_map(org.apache.flink.api.common.functions.FlatMapFunction<org.python.core.PyObject,Object> flat_mapper) A thin wrapper layer over DataStream.flatMap(FlatMapFunction).
PythonIterativeStream	iterate() A thin wrapper layer over DataStream.iterate().
PythonIterativeStream	iterate(Long max_wait_time_ms) A thin wrapper layer over DataStream.iterate(long).
PythonKeyedStream	key_by(org.apache.flink.api.java.functions.KeySelector<org.python.core.PyObject,PyKey> selector) A thin wrapper layer over DataStream.keyBy(KeySelector).
PythonDataStream<org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator<org.python.core.PyObject>>	map(org.apache.flink.api.common.functions.MapFunction<org.python.core.PyObject,org.python.core.PyObject> mapper) A thin wrapper layer over DataStream.map(MapFunction).
void	output() A thin wrapper layer over DataStream.print().
PythonSplitStream	split(org.apache.flink.streaming.api.collector.selector.OutputSelector<org.python.core.PyObject> output_selector) A thin wrapper layer over DataStream.split(OutputSelector).
PythonDataStream	union(PythonDataStream... streams) A thin wrapper layer over DataStream.union(DataStream[]).
void	write_as_text(String path) A thin wrapper layer over DataStream.writeAsText(java.lang.String).
void	write_as_text(String path, org.apache.flink.core.fs.FileSystem.WriteMode mode) A thin wrapper layer over DataStream#writeAsText(java.lang.String, WriteMode).
void	write_to_socket(String host, Integer port, org.apache.flink.api.common.serialization.SerializationSchema<org.python.core.PyObject> schema) A thin wrapper layer over DataStream.writeToSocket(String, int, org.apache.flink.api.common.serialization.SerializationSchema).

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

stream

protected final D extends org.apache.flink.streaming.api.datastream.DataStream<org.python.core.PyObject> stream

Constructor Detail

PythonDataStream

public PythonDataStream(D stream)

Method Detail

union

@SafeVarargs
public final PythonDataStream union(PythonDataStream... streams)

A thin wrapper layer over DataStream.union(DataStream[]).

Parameters:

streams - The Python DataStreams to union output with.

Returns:

The PythonDataStream.

split

public PythonSplitStream split(org.apache.flink.streaming.api.collector.selector.OutputSelector<org.python.core.PyObject> output_selector)
                        throws IOException

A thin wrapper layer over DataStream.split(OutputSelector).

Parameters:

output_selector - The user defined OutputSelector for directing the tuples.

Returns:

The PythonSplitStream

Throws:

IOException

filter

public PythonSingleOutputStreamOperator filter(org.apache.flink.api.common.functions.FilterFunction<org.python.core.PyObject> filter)
                                        throws IOException

A thin wrapper layer over DataStream.filter(FilterFunction).

Parameters:

filter - The FilterFunction that is called for each element of the DataStream.

Returns:

The filtered PythonDataStream.

Throws:

IOException

map

public PythonDataStream<org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator<org.python.core.PyObject>> map(org.apache.flink.api.common.functions.MapFunction<org.python.core.PyObject,org.python.core.PyObject> mapper)
                                                                                                                     throws IOException

A thin wrapper layer over DataStream.map(MapFunction).

Parameters:

mapper - The MapFunction that is called for each element of the DataStream.

Returns:

The transformed PythonDataStream.

Throws:

IOException

flat_map

public PythonDataStream<org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator<org.python.core.PyObject>> flat_map(org.apache.flink.api.common.functions.FlatMapFunction<org.python.core.PyObject,Object> flat_mapper)
                                                                                                                          throws IOException

A thin wrapper layer over DataStream.flatMap(FlatMapFunction).

Parameters:

flat_mapper - The FlatMapFunction that is called for each element of the DataStream

Returns:

The transformed PythonDataStream.

Throws:

IOException

key_by

public PythonKeyedStream key_by(org.apache.flink.api.java.functions.KeySelector<org.python.core.PyObject,PyKey> selector)
                         throws IOException

A thin wrapper layer over DataStream.keyBy(KeySelector).

Parameters:

selector - The KeySelector to be used for extracting the key for partitioning

Returns:

The PythonDataStream with partitioned state (i.e. PythonKeyedStream)

Throws:

IOException

output

@PublicEvolving
public void output()

A thin wrapper layer over DataStream.print().

write_as_text

@PublicEvolving
public void write_as_text(String path)

A thin wrapper layer over DataStream.writeAsText(java.lang.String).

Parameters:

path - The path pointing to the location the text file is written to.

write_as_text

@PublicEvolving
public void write_as_text(String path,
                                          org.apache.flink.core.fs.FileSystem.WriteMode mode)

A thin wrapper layer over DataStream#writeAsText(java.lang.String, WriteMode).

Parameters:

path - The path pointing to the location the text file is written to

mode - Controls the behavior for existing files. Options are NO_OVERWRITE and OVERWRITE.

write_to_socket

@PublicEvolving
public void write_to_socket(String host,
                                            Integer port,
                                            org.apache.flink.api.common.serialization.SerializationSchema<org.python.core.PyObject> schema)
                                     throws IOException

A thin wrapper layer over DataStream.writeToSocket(String, int, org.apache.flink.api.common.serialization.SerializationSchema).

Parameters:

host - host of the socket

port - port of the socket

schema - schema for serialization

Throws:

IOException

add_sink

@PublicEvolving
public void add_sink(org.apache.flink.streaming.api.functions.sink.SinkFunction<org.python.core.PyObject> sink_func)
                              throws IOException

A thin wrapper layer over DataStream.addSink(SinkFunction).

Parameters:

sink_func - The object containing the sink's invoke function.

Throws:

IOException

iterate

@PublicEvolving
public PythonIterativeStream iterate()

A thin wrapper layer over DataStream.iterate().

Initiates an iterative part of the program that feeds back data streams. The iterative part needs to be closed by calling PythonIterativeStream.close_with(PythonDataStream). The transformation of this IterativeStream will be the iteration head. The data stream given to the PythonIterativeStream.close_with(PythonDataStream) method is the data stream that will be fed back and used as the input for the iteration head.

A common usage pattern for streaming iterations is to use output splitting to send a part of the closing data stream to the head. Refer to split(OutputSelector) for more information.

The iteration edge will be partitioned the same way as the first input of the iteration head unless it is changed in the PythonIterativeStream.close_with(PythonDataStream) call.

By default a PythonDataStream with iteration will never terminate, but the user can use the maxWaitTime parameter to set a max waiting time for the iteration head. If no data received in the set time, the stream terminates.

Returns:

The iterative data stream created.

iterate

@PublicEvolving
public PythonIterativeStream iterate(Long max_wait_time_ms)

A thin wrapper layer over DataStream.iterate(long).

Initiates an iterative part of the program that feeds back data streams. The iterative part needs to be closed by calling PythonIterativeStream.close_with(PythonDataStream). The transformation of this IterativeStream will be the iteration head. The data stream given to the PythonIterativeStream.close_with(PythonDataStream) method is the data stream that will be fed back and used as the input for the iteration head.

A common usage pattern for streaming iterations is to use output splitting to send a part of the closing data stream to the head. Refer to split(OutputSelector) for more information.

The iteration edge will be partitioned the same way as the first input of the iteration head unless it is changed in the PythonIterativeStream.close_with(PythonDataStream) call.

By default a PythonDataStream with iteration will never terminate, but the user can use the maxWaitTime parameter to set a max waiting time for the iteration head. If no data received in the set time, the stream terminates.

Parameters:

max_wait_time_ms - Number of milliseconds to wait between inputs before shutting down

Returns:

The iterative data stream created.