Type
- The type of the messages created by the source.UId
- The type of unique IDs which may be used to acknowledge elements.@PublicEvolving public abstract class MessageAcknowledgingSourceBase<Type,UId> extends RichSourceFunction<Type> implements CheckpointedFunction, org.apache.flink.runtime.state.CheckpointListener
The mechanism for this source assumes that messages are identified by a unique ID. When messages are taken from the message queue, the message must not be dropped immediately, but must be retained until acknowledged. Messages that are not acknowledged within a certain time interval will be served again (to a different connection, established by the recovered source).
Note that this source can give no guarantees about message order in the case of failures, because messages that were retrieved but not yet acknowledged will be returned later again, after a set of messages that was not retrieved before the failure.
Internally, this source gathers the IDs of elements it emits. Per checkpoint, the IDs are stored and acknowledged when the checkpoint is complete. That way, no message is acknowledged unless it is certain that it has been successfully processed throughout the topology and the updates to any state caused by that message are persistent.
All messages that are emitted and successfully processed by the streaming program will eventually be acknowledged. In corner cases, the source may receive certain IDs multiple times, if a failure occurs while acknowledging. To cope with this situation, an additional Set stores all processed IDs. IDs are only removed after they have been acknowledged.
A typical way to use this base in a source function is by implementing a run() method as follows:
public void run(SourceContext<Type> ctx) throws Exception {
while (running) {
Message msg = queue.retrieve();
synchronized (ctx.getCheckpointLock()) {
if (addId(msg.getMessageId())) {
ctx.collect(msg.getMessageData());
}
}
}
}
NOTE: This source has a parallelism of 1
.SourceFunction.SourceContext<T>
限定符和类型 | 字段和说明 |
---|---|
protected ArrayDeque<org.apache.flink.api.java.tuple.Tuple2<Long,Set<UId>>> |
pendingCheckpoints
The list with IDs from checkpoints that were triggered, but not yet completed or notified of
completion.
|
限定符 | 构造器和说明 |
---|---|
protected |
MessageAcknowledgingSourceBase(Class<UId> idClass)
Creates a new MessageAcknowledgingSourceBase for IDs of the given type.
|
protected |
MessageAcknowledgingSourceBase(org.apache.flink.api.common.typeinfo.TypeInformation<UId> idTypeInfo)
Creates a new MessageAcknowledgingSourceBase for IDs of the given type.
|
限定符和类型 | 方法和说明 |
---|---|
protected abstract void |
acknowledgeIDs(long checkpointId,
Set<UId> uIds)
This method must be implemented to acknowledge the given set of IDs back to the message queue.
|
protected boolean |
addId(UId uid)
Adds an ID to be stored with the current checkpoint.
|
void |
close() |
void |
initializeState(org.apache.flink.runtime.state.FunctionInitializationContext context)
This method is called when the parallel function instance is created during distributed
execution.
|
void |
notifyCheckpointComplete(long checkpointId) |
void |
snapshotState(org.apache.flink.runtime.state.FunctionSnapshotContext context)
This method is called when a snapshot for a checkpoint is requested.
|
getIterationRuntimeContext, getRuntimeContext, open, setRuntimeContext
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
cancel, run
protected transient ArrayDeque<org.apache.flink.api.java.tuple.Tuple2<Long,Set<UId>>> pendingCheckpoints
protected MessageAcknowledgingSourceBase(Class<UId> idClass)
idClass
- The class of the message ID type, used to create a serializer for the message IDs.protected MessageAcknowledgingSourceBase(org.apache.flink.api.common.typeinfo.TypeInformation<UId> idTypeInfo)
idTypeInfo
- The type information of the message ID type, used to create a serializer for the message IDs.public void initializeState(org.apache.flink.runtime.state.FunctionInitializationContext context) throws Exception
CheckpointedFunction
initializeState
在接口中 CheckpointedFunction
context
- the context for initializing the operatorException
public void close() throws Exception
close
在接口中 org.apache.flink.api.common.functions.RichFunction
close
在类中 org.apache.flink.api.common.functions.AbstractRichFunction
Exception
protected abstract void acknowledgeIDs(long checkpointId, Set<UId> uIds)
uIds
- The list od IDs to acknowledge.protected boolean addId(UId uid)
uid
- The ID to add.public void snapshotState(org.apache.flink.runtime.state.FunctionSnapshotContext context) throws Exception
CheckpointedFunction
FunctionInitializationContext
when
the Function was initialized, or offered now by FunctionSnapshotContext
itself.snapshotState
在接口中 CheckpointedFunction
context
- the context for drawing a snapshot of the operatorException
Copyright © 2014–2019 The Apache Software Foundation. All rights reserved.