Package io.pravega.connectors.flink

Class FlinkPravegaOutputFormat<T>

java.lang.Object

org.apache.flink.api.common.io.RichOutputFormat<T>

io.pravega.connectors.flink.FlinkPravegaOutputFormat<T>

All Implemented Interfaces:

java.io.Serializable, org.apache.flink.api.common.io.OutputFormat<T>

public class FlinkPravegaOutputFormat<T>
extends org.apache.flink.api.common.io.RichOutputFormat<T>

A Flink OutputFormat that can be added as a sink to write into Pravega. The current implementation does not support transactional writes. The FlinkPravegaOutputFormat is used in FlinkPravegaTableSink implementation to support writing to Pravega as part of batch DataSet operation. See io.pravega.connectors.flink.FlinkTableITCase for more details on how to use it.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	FlinkPravegaOutputFormat.Builder<T>

Constructor Summary

Constructors

Constructor	Description
FlinkPravegaOutputFormat(io.pravega.client.ClientConfig clientConfig, io.pravega.client.stream.Stream stream, org.apache.flink.api.common.serialization.SerializationSchema<T> serializationSchema, PravegaEventRouter<T> eventRouter)	Creates a new Flink Pravega OutputFormat which can be added as a sink to a Flink batch job.

Method Summary

Modifier and Type	Method	Description
static <T> FlinkPravegaOutputFormat.Builder<T>	builder()
protected void	checkWriteError()
void	close()
void	configure(org.apache.flink.configuration.Configuration parameters)
protected io.pravega.client.EventStreamClientFactory	createClientFactory(java.lang.String scopeName, io.pravega.client.ClientConfig clientConfig)
protected java.util.concurrent.ExecutorService	createExecutorService()
protected PravegaEventRouter<T>	getEventRouter()
protected java.util.concurrent.atomic.AtomicInteger	getPendingWritesCount()
protected java.lang.String	getScope()
protected org.apache.flink.api.common.serialization.SerializationSchema<T>	getSerializationSchema()
protected java.lang.String	getStream()
protected boolean	isErrorOccurred()
void	open(int taskNumber, int numTasks)
void	writeRecord(T record)

Methods inherited from class org.apache.flink.api.common.io.RichOutputFormat

getRuntimeContext, setRuntimeContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

FlinkPravegaOutputFormat

public FlinkPravegaOutputFormat(io.pravega.client.ClientConfig clientConfig,
                                io.pravega.client.stream.Stream stream,
                                org.apache.flink.api.common.serialization.SerializationSchema<T> serializationSchema,
                                PravegaEventRouter<T> eventRouter)

Creates a new Flink Pravega OutputFormat which can be added as a sink to a Flink batch job.

Parameters:

clientConfig - The Pravega client configuration.

stream - The stream to write the events.

serializationSchema - The implementation to serialize events that will be written to pravega stream.

eventRouter - The event router to be used while writing the events.

Method Detail

configure

public void configure(org.apache.flink.configuration.Configuration parameters)

open

public void open(int taskNumber,
                 int numTasks)
          throws java.io.IOException

Throws:

java.io.IOException

writeRecord

public void writeRecord(T record)
                 throws java.io.IOException

Throws:

java.io.IOException

close

public void close()
           throws java.io.IOException

Throws:

java.io.IOException

checkWriteError

@VisibleForTesting
protected void checkWriteError()
                        throws java.io.IOException

Throws:

java.io.IOException

createClientFactory

@VisibleForTesting
protected io.pravega.client.EventStreamClientFactory createClientFactory(java.lang.String scopeName,
                                                                         io.pravega.client.ClientConfig clientConfig)

createExecutorService

@VisibleForTesting
protected java.util.concurrent.ExecutorService createExecutorService()

getSerializationSchema

@VisibleForTesting
protected org.apache.flink.api.common.serialization.SerializationSchema<T> getSerializationSchema()

getStream

@VisibleForTesting
protected java.lang.String getStream()

getScope

@VisibleForTesting
protected java.lang.String getScope()

isErrorOccurred

@VisibleForTesting
protected boolean isErrorOccurred()

getPendingWritesCount

@VisibleForTesting
protected java.util.concurrent.atomic.AtomicInteger getPendingWritesCount()

getEventRouter

@VisibleForTesting
protected PravegaEventRouter<T> getEventRouter()

builder

public static <T> FlinkPravegaOutputFormat.Builder<T> builder()