Package org.apache.flink.runtime.jobmaster

Class JobMaster

java.lang.Object

org.apache.flink.runtime.rpc.RpcEndpoint

org.apache.flink.runtime.rpc.FencedRpcEndpoint<JobMasterId>

org.apache.flink.runtime.jobmaster.JobMaster

All Implemented Interfaces:

AutoCloseable, BlocklistListener, CheckpointCoordinatorGateway, JobMasterGateway, JobMasterOperatorEventGateway, JobMasterService, KvStateLocationOracle, KvStateRegistryGateway, org.apache.flink.runtime.rpc.FencedRpcGateway<JobMasterId>, org.apache.flink.runtime.rpc.RpcGateway, org.apache.flink.util.AutoCloseableAsync

public class JobMaster
extends org.apache.flink.runtime.rpc.FencedRpcEndpoint<JobMasterId>
implements JobMasterGateway, JobMasterService

JobMaster implementation. The job master is responsible for the execution of a single ExecutionPlan.

It offers the following methods as part of its rpc interface to interact with the JobMaster remotely:

• updateTaskExecutionState(org.apache.flink.runtime.taskmanager.TaskExecutionState) updates the task execution state for given task

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.flink.runtime.rpc.RpcEndpoint

org.apache.flink.runtime.rpc.RpcEndpoint.MainThreadExecutor

Field Summary

Fields

Modifier and Type	Field	Description
static String	JOB_MANAGER_NAME	Default names for Flink's distributed components.

Fields inherited from class org.apache.flink.runtime.rpc.RpcEndpoint

log, rpcServer

Constructor Summary

Constructors

Constructor

Description

JobMaster(org.apache.flink.runtime.rpc.RpcService rpcService, JobMasterId jobMasterId, JobMasterConfiguration jobMasterConfiguration, ResourceID resourceId, ExecutionPlan executionPlan, HighAvailabilityServices highAvailabilityService, SlotPoolServiceSchedulerFactory slotPoolServiceSchedulerFactory, JobManagerSharedServices jobManagerSharedServices, HeartbeatServices heartbeatServices, JobManagerJobMetricGroupFactory jobMetricGroupFactory, OnCompletionActions jobCompletionActions, org.apache.flink.runtime.rpc.FatalErrorHandler fatalErrorHandler, ClassLoader userCodeLoader, ShuffleMaster<?> shuffleMaster, PartitionTrackerFactory partitionTrackerFactory, ExecutionDeploymentTracker executionDeploymentTracker, ExecutionDeploymentReconciler.Factory executionDeploymentReconcilerFactory, BlocklistHandler.Factory blocklistHandlerFactory, Collection<org.apache.flink.core.failure.FailureEnricher> failureEnrichers, long initializationTimestamp)

Method Summary

Modifier and Type	Method	Description
void	acknowledgeCheckpoint(org.apache.flink.api.common.JobID jobID, ExecutionAttemptID executionAttemptID, long checkpointId, CheckpointMetrics checkpointMetrics, org.apache.flink.util.SerializedValue<TaskStateSnapshot> checkpointState)
CompletableFuture<Acknowledge>	cancel(Duration timeout)	Cancels the currently executed job.
void	declineCheckpoint(DeclineCheckpoint decline)
CompletableFuture<CoordinationResponse>	deliverCoordinationRequestToCoordinator(OperatorID operatorId, org.apache.flink.util.SerializedValue<CoordinationRequest> serializedRequest, Duration timeout)	Deliver a coordination request to a specified coordinator and return the response.
void	disconnectResourceManager(ResourceManagerId resourceManagerId, Exception cause)	Disconnects the resource manager from the job manager because of the given cause.
CompletableFuture<Acknowledge>	disconnectTaskManager(ResourceID resourceID, Exception cause)	Disconnects the given TaskExecutor from the JobMaster.
void	failSlot(ResourceID taskManagerId, AllocationID allocationId, Exception cause)	Fails the slot with the given allocation id and cause.
JobMasterGateway	getGateway()	Get the JobMasterGateway belonging to this service.
CompletableFuture<Collection<PartitionWithMetrics>>	getPartitionWithMetrics(Duration timeout, Set<ResultPartitionID> expectedPartitions)	Get specified partitions and their metrics (identified by expectedPartitions), the metrics include sizes of sub-partitions in a result partition.
CompletableFuture<Void>	heartbeatFromResourceManager(ResourceID resourceID)	Sends heartbeat request from the resource manager.
CompletableFuture<Void>	heartbeatFromTaskManager(ResourceID resourceID, TaskExecutorToJobManagerHeartbeatPayload payload)	Sends the heartbeat to job manager from task manager.
void	notifyEndOfData(ExecutionAttemptID executionAttempt)	Notifies that the task has reached the end of data.
CompletableFuture<Acknowledge>	notifyKvStateRegistered(org.apache.flink.api.common.JobID jobId, JobVertexID jobVertexId, KeyGroupRange keyGroupRange, String registrationName, org.apache.flink.queryablestate.KvStateID kvStateId, InetSocketAddress kvStateServerAddress)	Notifies that queryable state has been registered.
CompletableFuture<Acknowledge>	notifyKvStateUnregistered(org.apache.flink.api.common.JobID jobId, JobVertexID jobVertexId, KeyGroupRange keyGroupRange, String registrationName)	Notifies that queryable state has been unregistered.
CompletableFuture<Acknowledge>	notifyNewBlockedNodes(Collection<BlockedNode> newNodes)	Notify new blocked node records.
void	notifyNotEnoughResourcesAvailable(Collection<ResourceRequirement> acquiredResources)	Notifies that not enough resources are available to fulfill the resource requirements of a job.
CompletableFuture<Collection<SlotOffer>>	offerSlots(ResourceID taskManagerId, Collection<SlotOffer> slots, Duration timeout)	Offers the given slots to the job manager.
protected void	onStart()
CompletableFuture<Void>	onStop()	Suspend the job and shutdown all other services including rpc.
CompletableFuture<RegistrationResponse>	registerTaskManager(org.apache.flink.api.common.JobID jobId, TaskManagerRegistrationInformation taskManagerRegistrationInformation, Duration timeout)	Registers the task manager at the job manager.
void	reportCheckpointMetrics(org.apache.flink.api.common.JobID jobID, ExecutionAttemptID executionAttemptID, long checkpointId, CheckpointMetrics checkpointMetrics)
void	reportInitializationMetrics(org.apache.flink.api.common.JobID jobId, ExecutionAttemptID executionAttemptId, SubTaskInitializationMetrics initializationMetrics)
CompletableFuture<CheckpointStatsSnapshot>	requestCheckpointStats(Duration timeout)	Requests the CheckpointStatsSnapshot of the job.
CompletableFuture<ExecutionGraphInfo>	requestJob(Duration timeout)	Requests the ExecutionGraphInfo of the executed job.
CompletableFuture<JobResourceRequirements>	requestJobResourceRequirements()	Read current job resource requirements.
CompletableFuture<org.apache.flink.api.common.JobStatus>	requestJobStatus(Duration timeout)	Requests the current job status.
CompletableFuture<KvStateLocation>	requestKvStateLocation(org.apache.flink.api.common.JobID jobId, String registrationName)	Requests a KvStateLocation for the specified InternalKvState registration name.
CompletableFuture<SerializedInputSplit>	requestNextInputSplit(JobVertexID vertexID, ExecutionAttemptID executionAttempt)	Requests the next input split for the ExecutionJobVertex.
CompletableFuture<ExecutionState>	requestPartitionState(IntermediateDataSetID intermediateResultId, ResultPartitionID resultPartitionId)	Requests the current state of the partition.
CompletableFuture<Acknowledge>	sendOperatorEventToCoordinator(ExecutionAttemptID task, OperatorID operatorID, org.apache.flink.util.SerializedValue<OperatorEvent> serializedEvent)
CompletableFuture<CoordinationResponse>	sendRequestToCoordinator(OperatorID operatorID, org.apache.flink.util.SerializedValue<CoordinationRequest> serializedRequest)
void	startFetchAndRetainPartitionWithMetricsOnTaskManager()	Notify jobMaster to fetch and retain partitions on task managers.
CompletableFuture<?>	stopTrackingAndReleasePartitions(Collection<ResultPartitionID> partitionIds)	Notifies the JobMasterPartitionTracker to stop tracking the target result partitions and release the locally occupied resources on TaskExecutors if any.
CompletableFuture<String>	stopWithSavepoint(String targetDirectory, org.apache.flink.core.execution.SavepointFormatType formatType, boolean terminate, Duration timeout)	Stops the job with a savepoint.
CompletableFuture<CompletedCheckpoint>	triggerCheckpoint(org.apache.flink.core.execution.CheckpointType checkpointType, Duration timeout)	Triggers taking a checkpoint of the executed job.
CompletableFuture<String>	triggerSavepoint(String targetDirectory, boolean cancelJob, org.apache.flink.core.execution.SavepointFormatType formatType, Duration timeout)	Triggers taking a savepoint of the executed job.
CompletableFuture<Object>	updateGlobalAggregate(String aggregateName, Object aggregand, byte[] serializedAggregateFunction)	Update the aggregate and return the new value.
CompletableFuture<Acknowledge>	updateJobResourceRequirements(JobResourceRequirements jobResourceRequirements)	Update job resource requirements.
CompletableFuture<Acknowledge>	updateTaskExecutionState(TaskExecutionState taskExecutionState)	Updates the task execution state for a given task.

Methods inherited from class org.apache.flink.runtime.rpc.FencedRpcEndpoint

getFencingToken

Methods inherited from class org.apache.flink.runtime.rpc.RpcEndpoint

callAsync, closeAsync, getAddress, getEndpointId, getHostname, getMainThreadExecutor, getMainThreadExecutor, getRpcService, getSelfGateway, getTerminationFuture, internalCallOnStart, internalCallOnStop, isRunning, registerResource, runAsync, scheduleRunAsync, scheduleRunAsync, start, stop, unregisterResource, validateRunsInMainThread

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.util.AutoCloseableAsync

close, closeAsync

Methods inherited from interface org.apache.flink.runtime.rpc.FencedRpcGateway

getFencingToken

Methods inherited from interface org.apache.flink.runtime.jobmaster.JobMasterGateway

triggerCheckpoint

Methods inherited from interface org.apache.flink.runtime.jobmaster.JobMasterService

getAddress, getTerminationFuture

Methods inherited from interface org.apache.flink.runtime.rpc.RpcGateway

getAddress, getHostname

Field Detail

JOB_MANAGER_NAME

public static final String JOB_MANAGER_NAME

Default names for Flink's distributed components.

See Also:

Constant Field Values

Constructor Detail

JobMaster

public JobMaster(org.apache.flink.runtime.rpc.RpcService rpcService,
                 JobMasterId jobMasterId,
                 JobMasterConfiguration jobMasterConfiguration,
                 ResourceID resourceId,
                 ExecutionPlan executionPlan,
                 HighAvailabilityServices highAvailabilityService,
                 SlotPoolServiceSchedulerFactory slotPoolServiceSchedulerFactory,
                 JobManagerSharedServices jobManagerSharedServices,
                 HeartbeatServices heartbeatServices,
                 JobManagerJobMetricGroupFactory jobMetricGroupFactory,
                 OnCompletionActions jobCompletionActions,
                 org.apache.flink.runtime.rpc.FatalErrorHandler fatalErrorHandler,
                 ClassLoader userCodeLoader,
                 ShuffleMaster<?> shuffleMaster,
                 PartitionTrackerFactory partitionTrackerFactory,
                 ExecutionDeploymentTracker executionDeploymentTracker,
                 ExecutionDeploymentReconciler.Factory executionDeploymentReconcilerFactory,
                 BlocklistHandler.Factory blocklistHandlerFactory,
                 Collection<org.apache.flink.core.failure.FailureEnricher> failureEnrichers,
                 long initializationTimestamp)
          throws Exception

Throws:

Exception

Method Detail

onStart

protected void onStart()
                throws JobMasterException

Overrides:

onStart in class org.apache.flink.runtime.rpc.RpcEndpoint

Throws:

JobMasterException

onStop

public CompletableFuture<Void> onStop()

Suspend the job and shutdown all other services including rpc.

Overrides:

onStop in class org.apache.flink.runtime.rpc.RpcEndpoint

cancel

public CompletableFuture<Acknowledge> cancel(Duration timeout)

Description copied from interface: JobMasterGateway

Cancels the currently executed job.

Specified by:

cancel in interface JobMasterGateway

Parameters:

timeout - of this operation

Returns:

Future acknowledge of the operation

updateTaskExecutionState

public CompletableFuture<Acknowledge> updateTaskExecutionState(TaskExecutionState taskExecutionState)

Updates the task execution state for a given task.

Specified by:

updateTaskExecutionState in interface JobMasterGateway

Parameters:

taskExecutionState - New task execution state for a given task

Returns:

Acknowledge the task execution state update

notifyEndOfData

public void notifyEndOfData(ExecutionAttemptID executionAttempt)

Description copied from interface: JobMasterGateway

Notifies that the task has reached the end of data.

Specified by:

notifyEndOfData in interface JobMasterGateway

Parameters:

executionAttempt - The execution attempt id.

requestNextInputSplit

public CompletableFuture<SerializedInputSplit> requestNextInputSplit(JobVertexID vertexID,
                                                                     ExecutionAttemptID executionAttempt)

Description copied from interface: JobMasterGateway

Requests the next input split for the ExecutionJobVertex. The next input split is sent back to the sender as a SerializedInputSplit message.

Specified by:

requestNextInputSplit in interface JobMasterGateway

Parameters:

vertexID - The job vertex id

executionAttempt - The execution attempt id

Returns:

The future of the input split. If there is no further input split, will return an empty object.

requestPartitionState

public CompletableFuture<ExecutionState> requestPartitionState(IntermediateDataSetID intermediateResultId,
                                                               ResultPartitionID resultPartitionId)

Description copied from interface: JobMasterGateway

Requests the current state of the partition. The state of a partition is currently bound to the state of the producing execution.

Specified by:

requestPartitionState in interface JobMasterGateway

Parameters:

intermediateResultId - The execution attempt ID of the task requesting the partition state.

resultPartitionId - The partition ID of the partition to request the state of.

Returns:

The future of the partition state

disconnectTaskManager

public CompletableFuture<Acknowledge> disconnectTaskManager(ResourceID resourceID,
                                                            Exception cause)

Description copied from interface: JobMasterGateway

Disconnects the given TaskExecutor from the JobMaster.

Specified by:

disconnectTaskManager in interface JobMasterGateway

Parameters:

resourceID - identifying the TaskManager to disconnect

cause - for the disconnection of the TaskManager

Returns:

Future acknowledge once the JobMaster has been disconnected from the TaskManager

acknowledgeCheckpoint

public void acknowledgeCheckpoint(org.apache.flink.api.common.JobID jobID,
                                  ExecutionAttemptID executionAttemptID,
                                  long checkpointId,
                                  CheckpointMetrics checkpointMetrics,
                                  @Nullable
                                  org.apache.flink.util.SerializedValue<TaskStateSnapshot> checkpointState)

Specified by:

acknowledgeCheckpoint in interface CheckpointCoordinatorGateway

reportCheckpointMetrics

public void reportCheckpointMetrics(org.apache.flink.api.common.JobID jobID,
                                    ExecutionAttemptID executionAttemptID,
                                    long checkpointId,
                                    CheckpointMetrics checkpointMetrics)

Specified by:

reportCheckpointMetrics in interface CheckpointCoordinatorGateway

reportInitializationMetrics

public void reportInitializationMetrics(org.apache.flink.api.common.JobID jobId,
                                        ExecutionAttemptID executionAttemptId,
                                        SubTaskInitializationMetrics initializationMetrics)

Specified by:

reportInitializationMetrics in interface CheckpointCoordinatorGateway

declineCheckpoint

public void declineCheckpoint(DeclineCheckpoint decline)

Specified by:

declineCheckpoint in interface CheckpointCoordinatorGateway

sendOperatorEventToCoordinator

public CompletableFuture<Acknowledge> sendOperatorEventToCoordinator(ExecutionAttemptID task,
                                                                     OperatorID operatorID,
                                                                     org.apache.flink.util.SerializedValue<OperatorEvent> serializedEvent)

Specified by:

sendOperatorEventToCoordinator in interface JobMasterOperatorEventGateway

sendRequestToCoordinator

public CompletableFuture<CoordinationResponse> sendRequestToCoordinator(OperatorID operatorID,
                                                                        org.apache.flink.util.SerializedValue<CoordinationRequest> serializedRequest)

Specified by:

sendRequestToCoordinator in interface JobMasterOperatorEventGateway

requestKvStateLocation

public CompletableFuture<KvStateLocation> requestKvStateLocation(org.apache.flink.api.common.JobID jobId,
                                                                 String registrationName)

Description copied from interface: KvStateLocationOracle

Requests a KvStateLocation for the specified InternalKvState registration name.

Specified by:

requestKvStateLocation in interface KvStateLocationOracle

Parameters:

jobId - identifying the job for which to request the KvStateLocation

registrationName - Name under which the KvState has been registered.

Returns:

Future of the requested InternalKvState location

notifyKvStateRegistered

public CompletableFuture<Acknowledge> notifyKvStateRegistered(org.apache.flink.api.common.JobID jobId,
                                                              JobVertexID jobVertexId,
                                                              KeyGroupRange keyGroupRange,
                                                              String registrationName,
                                                              org.apache.flink.queryablestate.KvStateID kvStateId,
                                                              InetSocketAddress kvStateServerAddress)

Description copied from interface: KvStateRegistryGateway

Notifies that queryable state has been registered.

Specified by:

notifyKvStateRegistered in interface KvStateRegistryGateway

Parameters:

jobId - identifying the job for which to register a key value state

jobVertexId - JobVertexID the KvState instance belongs to.

keyGroupRange - Key group range the KvState instance belongs to.

registrationName - Name under which the KvState has been registered.

kvStateId - ID of the registered KvState instance.

kvStateServerAddress - Server address where to find the KvState instance.

Returns:

Future acknowledge if the key-value state has been registered

notifyKvStateUnregistered

public CompletableFuture<Acknowledge> notifyKvStateUnregistered(org.apache.flink.api.common.JobID jobId,
                                                                JobVertexID jobVertexId,
                                                                KeyGroupRange keyGroupRange,
                                                                String registrationName)

Description copied from interface: KvStateRegistryGateway

Notifies that queryable state has been unregistered.

Specified by:

notifyKvStateUnregistered in interface KvStateRegistryGateway

Parameters:

jobId - identifying the job for which to unregister a key value state

jobVertexId - JobVertexID the KvState instance belongs to.

keyGroupRange - Key group index the KvState instance belongs to.

registrationName - Name under which the KvState has been registered.

Returns:

Future acknowledge if the key-value state has been unregistered

offerSlots

public CompletableFuture<Collection<SlotOffer>> offerSlots(ResourceID taskManagerId,
                                                           Collection<SlotOffer> slots,
                                                           Duration timeout)

Description copied from interface: JobMasterGateway

Offers the given slots to the job manager. The response contains the set of accepted slots.

Specified by:

offerSlots in interface JobMasterGateway

Parameters:

taskManagerId - identifying the task manager

slots - to offer to the job manager

timeout - for the rpc call

Returns:

Future set of accepted slots.

failSlot

public void failSlot(ResourceID taskManagerId,
                     AllocationID allocationId,
                     Exception cause)

Description copied from interface: JobMasterGateway

Fails the slot with the given allocation id and cause.

Specified by:

failSlot in interface JobMasterGateway

Parameters:

taskManagerId - identifying the task manager

allocationId - identifying the slot to fail

cause - of the failing

registerTaskManager

public CompletableFuture<RegistrationResponse> registerTaskManager(org.apache.flink.api.common.JobID jobId,
                                                                   TaskManagerRegistrationInformation taskManagerRegistrationInformation,
                                                                   Duration timeout)

Description copied from interface: JobMasterGateway

Registers the task manager at the job manager.

Specified by:

registerTaskManager in interface JobMasterGateway

Parameters:

jobId - jobId specifying the job for which the JobMaster should be responsible

taskManagerRegistrationInformation - the information for registering a task manager at the job manager

timeout - for the rpc call

Returns:

Future registration response indicating whether the registration was successful or not

disconnectResourceManager

public void disconnectResourceManager(ResourceManagerId resourceManagerId,
                                      Exception cause)

Description copied from interface: JobMasterGateway

Disconnects the resource manager from the job manager because of the given cause.

Specified by:

disconnectResourceManager in interface JobMasterGateway

Parameters:

resourceManagerId - identifying the resource manager leader id

cause - of the disconnect

heartbeatFromTaskManager

public CompletableFuture<Void> heartbeatFromTaskManager(ResourceID resourceID,
                                                        TaskExecutorToJobManagerHeartbeatPayload payload)

Description copied from interface: JobMasterGateway

Sends the heartbeat to job manager from task manager.

Specified by:

heartbeatFromTaskManager in interface JobMasterGateway

Parameters:

resourceID - unique id of the task manager

payload - report payload

Returns:

future which is completed exceptionally if the operation fails

heartbeatFromResourceManager

public CompletableFuture<Void> heartbeatFromResourceManager(ResourceID resourceID)

Description copied from interface: JobMasterGateway

Sends heartbeat request from the resource manager.

Specified by:

heartbeatFromResourceManager in interface JobMasterGateway

Parameters:

resourceID - unique id of the resource manager

Returns:

future which is completed exceptionally if the operation fails

requestJobStatus

public CompletableFuture<org.apache.flink.api.common.JobStatus> requestJobStatus(Duration timeout)

Description copied from interface: JobMasterGateway

Requests the current job status.

Specified by:

requestJobStatus in interface JobMasterGateway

Parameters:

timeout - for the rpc call

Returns:

Future containing the current job status

requestJob

public CompletableFuture<ExecutionGraphInfo> requestJob(Duration timeout)

Description copied from interface: JobMasterGateway

Requests the ExecutionGraphInfo of the executed job.

Specified by:

requestJob in interface JobMasterGateway

Parameters:

timeout - for the rpc call

Returns:

Future which is completed with the ExecutionGraphInfo of the executed job

requestCheckpointStats

public CompletableFuture<CheckpointStatsSnapshot> requestCheckpointStats(Duration timeout)

Description copied from interface: JobMasterGateway

Requests the CheckpointStatsSnapshot of the job.

Specified by:

requestCheckpointStats in interface JobMasterGateway

Parameters:

timeout - for the rpc call

Returns:

Future which is completed with the CheckpointStatsSnapshot of the job

triggerCheckpoint

public CompletableFuture<CompletedCheckpoint> triggerCheckpoint(org.apache.flink.core.execution.CheckpointType checkpointType,
                                                                Duration timeout)

Description copied from interface: JobMasterGateway

Triggers taking a checkpoint of the executed job.

Specified by:

triggerCheckpoint in interface JobMasterGateway

Parameters:

checkpointType - to determine how checkpoint should be taken

timeout - for the rpc call

Returns:

Future which is completed with the CompletedCheckpoint once completed

triggerSavepoint

public CompletableFuture<String> triggerSavepoint(@Nullable
                                                  String targetDirectory,
                                                  boolean cancelJob,
                                                  org.apache.flink.core.execution.SavepointFormatType formatType,
                                                  Duration timeout)

Description copied from interface: JobMasterGateway

Triggers taking a savepoint of the executed job.

Specified by:

triggerSavepoint in interface JobMasterGateway

Parameters:

targetDirectory - to which to write the savepoint data or null if the default savepoint directory should be used

formatType - binary format for the savepoint

timeout - for the rpc call

Returns:

Future which is completed with the savepoint path once completed

stopWithSavepoint

public CompletableFuture<String> stopWithSavepoint(@Nullable
                                                   String targetDirectory,
                                                   org.apache.flink.core.execution.SavepointFormatType formatType,
                                                   boolean terminate,
                                                   Duration timeout)

Description copied from interface: JobMasterGateway

Stops the job with a savepoint.

Specified by:

stopWithSavepoint in interface JobMasterGateway

Parameters:

targetDirectory - to which to write the savepoint data or null if the default savepoint directory should be used

terminate - flag indicating if the job should terminate or just suspend

timeout - for the rpc call

Returns:

Future which is completed with the savepoint path once completed

notifyNotEnoughResourcesAvailable

public void notifyNotEnoughResourcesAvailable(Collection<ResourceRequirement> acquiredResources)

Description copied from interface: JobMasterGateway

Notifies that not enough resources are available to fulfill the resource requirements of a job.

Specified by:

notifyNotEnoughResourcesAvailable in interface JobMasterGateway

Parameters:

acquiredResources - the resources that have been acquired for the job

updateGlobalAggregate

public CompletableFuture<Object> updateGlobalAggregate(String aggregateName,
                                                       Object aggregand,
                                                       byte[] serializedAggregateFunction)

Description copied from interface: JobMasterGateway

Update the aggregate and return the new value.

Specified by:

updateGlobalAggregate in interface JobMasterGateway

Parameters:

aggregateName - The name of the aggregate to update

aggregand - The value to add to the aggregate

serializedAggregateFunction - The function to apply to the current aggregate and aggregand to obtain the new aggregate value, this should be of type AggregateFunction

Returns:

The updated aggregate

deliverCoordinationRequestToCoordinator

public CompletableFuture<CoordinationResponse> deliverCoordinationRequestToCoordinator(OperatorID operatorId,
                                                                                       org.apache.flink.util.SerializedValue<CoordinationRequest> serializedRequest,
                                                                                       Duration timeout)

Description copied from interface: JobMasterGateway

Deliver a coordination request to a specified coordinator and return the response.

Specified by:

deliverCoordinationRequestToCoordinator in interface JobMasterGateway

Parameters:

operatorId - identifying the coordinator to receive the request

serializedRequest - serialized request to deliver

Returns:

A future containing the response. The response will fail with a FlinkException if the task is not running, or no operator/coordinator exists for the given ID, or the coordinator cannot handle client events.

stopTrackingAndReleasePartitions

public CompletableFuture<?> stopTrackingAndReleasePartitions(Collection<ResultPartitionID> partitionIds)

Description copied from interface: JobMasterGateway

Notifies the JobMasterPartitionTracker to stop tracking the target result partitions and release the locally occupied resources on TaskExecutors if any.

Specified by:

stopTrackingAndReleasePartitions in interface JobMasterGateway

getPartitionWithMetrics

public CompletableFuture<Collection<PartitionWithMetrics>> getPartitionWithMetrics(Duration timeout,
                                                                                   Set<ResultPartitionID> expectedPartitions)

Description copied from interface: JobMasterGateway

Get specified partitions and their metrics (identified by expectedPartitions), the metrics include sizes of sub-partitions in a result partition.

Specified by:

getPartitionWithMetrics in interface JobMasterGateway

Parameters:

timeout - The timeout used for retrieve the specified partitions.

expectedPartitions - The set of identifiers for the result partitions whose metrics are to be fetched.

Returns:

A future will contain a collection of the partitions with their metrics that could be retrieved from the expected partitions within the specified timeout period.

startFetchAndRetainPartitionWithMetricsOnTaskManager

public void startFetchAndRetainPartitionWithMetricsOnTaskManager()

Description copied from interface: JobMasterGateway

Notify jobMaster to fetch and retain partitions on task managers. It will process for future TaskManager registrations and already registered TaskManagers.

Specified by:

startFetchAndRetainPartitionWithMetricsOnTaskManager in interface JobMasterGateway

notifyNewBlockedNodes

public CompletableFuture<Acknowledge> notifyNewBlockedNodes(Collection<BlockedNode> newNodes)

Description copied from interface: BlocklistListener

Notify new blocked node records.

Specified by:

notifyNewBlockedNodes in interface BlocklistListener

Parameters:

newNodes - the new blocked node records

Returns:

Future acknowledge once the new nodes have successfully notified.

requestJobResourceRequirements

public CompletableFuture<JobResourceRequirements> requestJobResourceRequirements()

Description copied from interface: JobMasterGateway

Read current job resource requirements.

Specified by:

requestJobResourceRequirements in interface JobMasterGateway

Returns:

Future which that contains current resource requirements.

updateJobResourceRequirements

public CompletableFuture<Acknowledge> updateJobResourceRequirements(JobResourceRequirements jobResourceRequirements)

Description copied from interface: JobMasterGateway

Update job resource requirements.

Specified by:

updateJobResourceRequirements in interface JobMasterGateway

Parameters:

jobResourceRequirements - new resource requirements

Returns:

Future which is completed successfully when requirements are updated

getGateway

public JobMasterGateway getGateway()

Description copied from interface: JobMasterService

Get the JobMasterGateway belonging to this service.

Specified by:

getGateway in interface JobMasterService

Returns:

JobMasterGateway belonging to this service