Package org.apache.flink.runtime.asyncprocessing

Class AsyncExecutionController<K,REQUEST extends AsyncRequest<?>>

java.lang.Object

org.apache.flink.runtime.asyncprocessing.AsyncExecutionController<K,REQUEST>

Type Parameters:

K - the type of the key

All Implemented Interfaces:

Closeable, AutoCloseable

Direct Known Subclasses:

SimpleAsyncExecutionController, StateExecutionController

public class AsyncExecutionController<K,REQUEST extends AsyncRequest<?>>
extends Object
implements Closeable

The Async Execution Controller (AEC) receives processing requests from operators, and put them into execution according to some strategies.

It is responsible for:

Preserving the sequence of elements bearing the same key by delaying subsequent requests until the processing of preceding ones is finalized. Tracking the in-flight data(records) and blocking the input if too much data in flight (back-pressure). It invokes MailboxExecutor.yield() to pause current operations, allowing for the execution of callbacks (mails in Mailbox).

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static interface	AsyncExecutionController.SwitchContextListener<K>	A listener listens the key context switch.

Constructor Summary

Constructors

Constructor

Description

AsyncExecutionController(org.apache.flink.api.common.operators.MailboxExecutor mailboxExecutor, org.apache.flink.core.asyncprocessing.AsyncFutureImpl.AsyncFrameworkExceptionHandler exceptionHandler, AsyncExecutor<REQUEST> asyncExecutor, DeclarationManager declarationManager, EpochManager.ParallelMode epochParallelMode, int maxParallelism, int batchSize, long bufferTimeout, int maxInFlightRecords, AsyncExecutionController.SwitchContextListener<K> switchContextListener, org.apache.flink.metrics.MetricGroup metricGroup)

Method Summary

Modifier and Type	Method	Description
RecordContext<K>	buildContext(Object record, K key)	Build a new context based on record and key.
RecordContext<K>	buildContext(Object record, K key, boolean inherit)	Build a new context based on record and key.
void	close()
void	drainInflightRecords(int targetNum)	A helper function to drain in-flight records util inFlightRecordNum within the limit of given targetNum.
AsyncExecutor<REQUEST>	getAsyncExecutor()
RecordContext<K>	getCurrentContext()
int	getInFlightRecordNum()
org.apache.flink.api.common.operators.MailboxExecutor	getMailboxExecutor()
void	handleRequest(AsyncRequest<K> request, boolean allowOverdraft)	Submit a AsyncRequest to this AsyncExecutionController and trigger it if needed.
void	processNonRecord(org.apache.flink.util.function.ThrowingRunnable<? extends Exception> triggerAction, org.apache.flink.util.function.ThrowingRunnable<? extends Exception> finalAction)
void	setCurrentContext(RecordContext<K> switchingContext)	Each time before a code segment (callback) is about to run in mailbox (task thread), this method should be called to switch a context in AEC.
org.apache.flink.core.asyncprocessing.AsyncFuture<Void>	syncPointRequestWithCallback(org.apache.flink.util.function.ThrowingRunnable<Exception> callback, boolean allowOverdraft)	A helper to request a sync point and run a callback if it finishes (once the record is not blocked).
boolean	triggerIfNeeded(boolean force)	Trigger a batch of requests.
void	waitUntil(Supplier<Boolean> checker)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

AsyncExecutionController

public AsyncExecutionController(org.apache.flink.api.common.operators.MailboxExecutor mailboxExecutor,
                                org.apache.flink.core.asyncprocessing.AsyncFutureImpl.AsyncFrameworkExceptionHandler exceptionHandler,
                                AsyncExecutor<REQUEST> asyncExecutor,
                                DeclarationManager declarationManager,
                                EpochManager.ParallelMode epochParallelMode,
                                int maxParallelism,
                                int batchSize,
                                long bufferTimeout,
                                int maxInFlightRecords,
                                @Nullable
                                AsyncExecutionController.SwitchContextListener<K> switchContextListener,
                                @Nullable
                                org.apache.flink.metrics.MetricGroup metricGroup)

Method Detail

buildContext

public RecordContext<K> buildContext(Object record,
                                     K key)

Build a new context based on record and key. Also wired with internal KeyAccountingUnit.

Parameters:

record - the given record.

key - the given key.

Returns:

the built record context.

buildContext

public RecordContext<K> buildContext(Object record,
                                     K key,
                                     boolean inherit)

Build a new context based on record and key. Also wired with internal KeyAccountingUnit.

Parameters:

record - the given record.

key - the given key.

inherit - whether to inherit epoch and variables from the current context. Or otherwise create new ones.

Returns:

the built record context.

setCurrentContext

public void setCurrentContext(RecordContext<K> switchingContext)

Each time before a code segment (callback) is about to run in mailbox (task thread), this method should be called to switch a context in AEC.

Parameters:

switchingContext - the context to switch.

getCurrentContext

public RecordContext<K> getCurrentContext()

handleRequest

public void handleRequest(AsyncRequest<K> request,
                          boolean allowOverdraft)

Submit a AsyncRequest to this AsyncExecutionController and trigger it if needed.

waitUntil

public void waitUntil(Supplier<Boolean> checker)

triggerIfNeeded

public boolean triggerIfNeeded(boolean force)

Trigger a batch of requests.

Parameters:

force - whether to trigger requests in force.

syncPointRequestWithCallback

public org.apache.flink.core.asyncprocessing.AsyncFuture<Void> syncPointRequestWithCallback(org.apache.flink.util.function.ThrowingRunnable<Exception> callback,
                                                                                            boolean allowOverdraft)

A helper to request a sync point and run a callback if it finishes (once the record is not blocked).

Parameters:

callback - the callback to run if it finishes (once the record is not blocked).

allowOverdraft - whether to overdraft the in-flight buffer.

drainInflightRecords

public void drainInflightRecords(int targetNum)

A helper function to drain in-flight records util inFlightRecordNum within the limit of given targetNum.

Parameters:

targetNum - the target inFlightRecordNum to achieve.

processNonRecord

public void processNonRecord(@Nullable
                             org.apache.flink.util.function.ThrowingRunnable<? extends Exception> triggerAction,
                             @Nullable
                             org.apache.flink.util.function.ThrowingRunnable<? extends Exception> finalAction)

getAsyncExecutor

@VisibleForTesting
public AsyncExecutor<REQUEST> getAsyncExecutor()

getInFlightRecordNum

@VisibleForTesting
public int getInFlightRecordNum()

getMailboxExecutor

@VisibleForTesting
public org.apache.flink.api.common.operators.MailboxExecutor getMailboxExecutor()

close

public void close()
           throws IOException

Specified by:

close in interface AutoCloseable

Specified by:

close in interface Closeable

Throws:

IOException