Class DefaultExecutionGraph
- java.lang.Object
-
- org.apache.flink.runtime.executiongraph.DefaultExecutionGraph
-
- All Implemented Interfaces:
AccessExecutionGraph,ExecutionGraph,InternalExecutionGraphAccessor,JobStatusProvider
public class DefaultExecutionGraph extends Object implements ExecutionGraph, InternalExecutionGraphAccessor
Default implementation of theExecutionGraph.
-
-
Constructor Summary
Constructors Constructor Description DefaultExecutionGraph(JobInformation jobInformation, ScheduledExecutorService futureExecutor, Executor ioExecutor, Duration rpcTimeout, int executionHistorySizeLimit, ClassLoader userClassLoader, BlobWriter blobWriter, PartitionGroupReleaseStrategy.Factory partitionGroupReleaseStrategyFactory, ShuffleMaster<?> shuffleMaster, JobMasterPartitionTracker partitionTracker, ExecutionDeploymentListener executionDeploymentListener, ExecutionStateUpdateListener executionStateUpdateListener, long initializationTimestamp, VertexAttemptNumberStore initialAttemptCounts, VertexParallelismStore vertexParallelismStore, boolean isDynamic, ExecutionJobVertex.Factory executionJobVertexFactory, List<org.apache.flink.core.execution.JobStatusHook> jobStatusHooks, MarkPartitionFinishedStrategy markPartitionFinishedStrategy, TaskDeploymentDescriptorFactory taskDeploymentDescriptorFactory, List<org.apache.flink.core.execution.JobStatusChangedListener> jobStatusChangedListeners, ExecutionPlanSchedulingContext executionPlanSchedulingContext)
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description voidaddNewJobVertices(List<JobVertex> topologicallySortedNewlyJobVertices, JobManagerJobMetricGroup jobManagerJobMetricGroup, VertexParallelismStore newVerticesParallelismStore)Adds new job vertices to the execution graph based on the provided list of topologically sorted job vertices.Map<String,org.apache.flink.util.OptionalFailure<org.apache.flink.api.common.accumulators.Accumulator<?,?>>>aggregateUserAccumulators()Merges all accumulator results from the tasks previously executed in the Executions.voidattachJobGraph(List<JobVertex> verticesToAttach, JobManagerJobMetricGroup jobManagerJobMetricGroup)voidcancel()ResultPartitionIDcreateResultPartitionId(IntermediateResultPartitionID resultPartitionId)voiddeleteBlobs(List<PermanentBlobKey> blobKeys)voidderegisterExecution(Execution exec)voidenableCheckpointing(CheckpointCoordinatorConfiguration chkConfig, List<MasterTriggerRestoreHook<?>> masterHooks, CheckpointIDCounter checkpointIDCounter, CompletedCheckpointStore checkpointStore, StateBackend checkpointStateBackend, CheckpointStorage checkpointStorage, CheckpointStatsTracker statsTracker, CheckpointsCleaner checkpointsCleaner, String changelogStorageName)voidfailGlobal(Throwable t)Fails the execution graph globally.voidfailJob(Throwable cause, long timestamp)Optional<AccessExecution>findExecution(ExecutionAttemptID attemptId)Optional<String>findVertexWithAttempt(ExecutionAttemptID attemptId)StringifiedAccumulatorResult[]getAccumulatorResultsStringified()Returns the a stringified version of the user-defined accumulators.Map<String,org.apache.flink.util.SerializedValue<org.apache.flink.util.OptionalFailure<Object>>>getAccumulatorsSerialized()Gets a serialized accumulator map.Iterable<ExecutionVertex>getAllExecutionVertices()Returns an iterable containing all execution vertices for this execution graph.Map<IntermediateDataSetID,IntermediateResult>getAllIntermediateResults()Map<JobVertexID,ExecutionJobVertex>getAllVertices()Returns a map containing all job vertices for this execution graph.org.apache.flink.api.common.ArchivedExecutionConfiggetArchivedExecutionConfig()Returns the serializableArchivedExecutionConfig.BlobWritergetBlobWriter()Optional<String>getChangelogStorageName()Returns the changelog storage name for this ExecutionGraph.CheckpointCoordinatorgetCheckpointCoordinator()CheckpointCoordinatorConfigurationgetCheckpointCoordinatorConfiguration()Returns theCheckpointCoordinatorConfigurationornullif checkpointing is disabled.CheckpointStatsSnapshotgetCheckpointStatsSnapshot()Returns a snapshot of the checkpoint statistics ornullif checkpointing is disabled.Optional<String>getCheckpointStorageName()Returns the checkpoint storage name for this ExecutionGraph.List<ShuffleDescriptor>getClusterPartitionShuffleDescriptors(IntermediateDataSetID intermediateDataSetID)Get the shuffle descriptors of the cluster partitions ordered by partition number.EdgeManagergetEdgeManager()ExecutionDeploymentListenergetExecutionDeploymentListener()ExecutionGraphIDgetExecutionGraphID()ExecutionVertexgetExecutionVertexOrThrow(ExecutionVertexID id)ThrowablegetFailureCause()ErrorInfogetFailureInfo()Returns the exception that caused the job to fail.ExecutorgetFutureExecutor()Returns the ExecutionContext associated with this ExecutionGraph.org.apache.flink.configuration.ConfigurationgetJobConfiguration()org.apache.flink.api.common.JobIDgetJobID()Returns theJobIDfor this execution graph.org.apache.flink.runtime.concurrent.ComponentMainThreadExecutorgetJobMasterMainThreadExecutor()StringgetJobName()Returns the job name for the execution graph.JobTypegetJobType()Returns theJobTypefor this execution graph.ExecutionJobVertexgetJobVertex(JobVertexID id)Returns the job vertex for the givenJobVertexID.JobVertexInputInfogetJobVertexInputInfo(JobVertexID jobVertexId, IntermediateDataSetID resultId)Get the input info of a certain input of a certain job vertex.KvStateLocationRegistrygetKvStateLocationRegistry()MarkPartitionFinishedStrategygetMarkPartitionFinishedStrategy()longgetNumberOfRestarts()Gets the number of restarts, including full restarts and fine grained restarts.intgetNumFinishedVertices()PartitionGroupReleaseStrategygetPartitionGroupReleaseStrategy()JobMasterPartitionTrackergetPartitionTracker()intgetPendingOperatorCount()Retrieves the count of pending operators waiting to be transferred to job vertices in the adaptive execution of batch jobs.JobPlanInfo.PlangetPlan()Returns the job plan as a JobPlanInfo.Plan.Map<ExecutionAttemptID,Execution>getRegisteredExecutions()ResultPartitionAvailabilityCheckergetResultPartitionAvailabilityChecker()IntermediateResultPartitiongetResultPartitionOrThrow(IntermediateResultPartitionID id)Gets the intermediate result partition by the given partition ID, or throw an exception if the partition is not found.SchedulingTopologygetSchedulingTopology()ShuffleMaster<?>getShuffleMaster()org.apache.flink.api.common.JobStatusgetState()Returns the currentJobStatusfor this execution graph.Optional<String>getStateBackendName()Returns the state backend name for this ExecutionGraph.longgetStatusTimestamp(org.apache.flink.api.common.JobStatus status)Returns the timestamp for the givenJobStatus.StringgetStreamGraphJson()Returns the stream graph as a JSON string.TaskDeploymentDescriptorFactorygetTaskDeploymentDescriptorFactory()CompletableFuture<org.apache.flink.api.common.JobStatus>getTerminationFuture()Returns the termination future of thisExecutionGraph.ClassLoadergetUserClassLoader()Iterable<ExecutionJobVertex>getVerticesTopologically()Returns an iterable containing all job vertices for this execution graph in the order they were created.voidincrementRestarts()voidinitFailureCause(Throwable t, long timestamp)voidinitializeJobVertex(ExecutionJobVertex ejv, long createTimestamp, Map<IntermediateDataSetID,JobVertexInputInfo> jobVertexInputInfos)Initialize the given execution job vertex, mainly includes creating execution vertices according to the parallelism, and connecting to the predecessors.org.apache.flink.util.TernaryBooleanisChangelogStateBackendEnabled()Returns whether the state changelog is enabled for this ExecutionGraph.booleanisDynamic()booleanisStoppable()Returns whether the job for this execution graph is stoppable.voidjobVertexFinished()Called whenever a job vertex reaches state FINISHED (completed successfully).voidjobVertexUnFinished()voidnotifyExecutionChange(Execution execution, ExecutionState previousState, ExecutionState newExecutionState)voidnotifyNewlyInitializedJobVertices(List<ExecutionJobVertex> vertices)Notify that some job vertices have been newly initialized, execution graph will try to update scheduling topology.voidnotifySchedulerNgAboutInternalTaskFailure(ExecutionAttemptID attemptId, Throwable t, boolean cancelTask, boolean releasePartitions)voidregisterExecution(Execution exec)voidregisterJobStatusListener(JobStatusListener listener)voidsetInternalTaskFailuresListener(InternalFailuresListener internalTaskFailuresListener)voidsetPlan(JobPlanInfo.Plan plan)voidstart(org.apache.flink.runtime.concurrent.ComponentMainThreadExecutor jobMasterMainThreadExecutor)voidsuspend(Throwable suspensionCause)Suspends the current ExecutionGraph.booleantransitionState(org.apache.flink.api.common.JobStatus current, org.apache.flink.api.common.JobStatus newState)voidtransitionToRunning()voidupdateAccumulators(AccumulatorSnapshot accumulatorSnapshot)Updates the accumulators during the runtime of a job.booleanupdateState(TaskExecutionStateTransition state)Updates the state of one of the ExecutionVertex's Execution attempts.org.apache.flink.api.common.JobStatuswaitUntilTerminal()-
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
-
Methods inherited from interface org.apache.flink.runtime.executiongraph.ExecutionGraph
initializeJobVertex
-
-
-
-
Constructor Detail
-
DefaultExecutionGraph
public DefaultExecutionGraph(JobInformation jobInformation, ScheduledExecutorService futureExecutor, Executor ioExecutor, Duration rpcTimeout, int executionHistorySizeLimit, ClassLoader userClassLoader, BlobWriter blobWriter, PartitionGroupReleaseStrategy.Factory partitionGroupReleaseStrategyFactory, ShuffleMaster<?> shuffleMaster, JobMasterPartitionTracker partitionTracker, ExecutionDeploymentListener executionDeploymentListener, ExecutionStateUpdateListener executionStateUpdateListener, long initializationTimestamp, VertexAttemptNumberStore initialAttemptCounts, VertexParallelismStore vertexParallelismStore, boolean isDynamic, ExecutionJobVertex.Factory executionJobVertexFactory, List<org.apache.flink.core.execution.JobStatusHook> jobStatusHooks, MarkPartitionFinishedStrategy markPartitionFinishedStrategy, TaskDeploymentDescriptorFactory taskDeploymentDescriptorFactory, List<org.apache.flink.core.execution.JobStatusChangedListener> jobStatusChangedListeners, ExecutionPlanSchedulingContext executionPlanSchedulingContext)
-
-
Method Detail
-
start
public void start(@Nonnull org.apache.flink.runtime.concurrent.ComponentMainThreadExecutor jobMasterMainThreadExecutor)- Specified by:
startin interfaceExecutionGraph
-
getSchedulingTopology
public SchedulingTopology getSchedulingTopology()
- Specified by:
getSchedulingTopologyin interfaceExecutionGraph
-
getJobMasterMainThreadExecutor
@Nonnull public org.apache.flink.runtime.concurrent.ComponentMainThreadExecutor getJobMasterMainThreadExecutor()
- Specified by:
getJobMasterMainThreadExecutorin interfaceExecutionGraph- Specified by:
getJobMasterMainThreadExecutorin interfaceInternalExecutionGraphAccessor
-
isChangelogStateBackendEnabled
public org.apache.flink.util.TernaryBoolean isChangelogStateBackendEnabled()
Description copied from interface:AccessExecutionGraphReturns whether the state changelog is enabled for this ExecutionGraph.- Specified by:
isChangelogStateBackendEnabledin interfaceAccessExecutionGraph- Returns:
- true, if state changelog enabled, false otherwise.
-
getStateBackendName
public Optional<String> getStateBackendName()
Description copied from interface:AccessExecutionGraphReturns the state backend name for this ExecutionGraph.- Specified by:
getStateBackendNamein interfaceAccessExecutionGraph- Returns:
- The state backend name, or an empty Optional in the case of batch jobs
-
getCheckpointStorageName
public Optional<String> getCheckpointStorageName()
Description copied from interface:AccessExecutionGraphReturns the checkpoint storage name for this ExecutionGraph.- Specified by:
getCheckpointStorageNamein interfaceAccessExecutionGraph- Returns:
- The checkpoint storage name, or an empty Optional in the case of batch jobs
-
getChangelogStorageName
public Optional<String> getChangelogStorageName()
Description copied from interface:AccessExecutionGraphReturns the changelog storage name for this ExecutionGraph.- Specified by:
getChangelogStorageNamein interfaceAccessExecutionGraph- Returns:
- The changelog storage name, or an empty Optional in the case of batch jobs
-
getPendingOperatorCount
public int getPendingOperatorCount()
Description copied from interface:AccessExecutionGraphRetrieves the count of pending operators waiting to be transferred to job vertices in the adaptive execution of batch jobs. This value will be zero if the job is submitted with a JobGraph or if it's a streaming job.- Specified by:
getPendingOperatorCountin interfaceAccessExecutionGraph- Returns:
- the number of pending operators.
-
enableCheckpointing
public void enableCheckpointing(CheckpointCoordinatorConfiguration chkConfig, List<MasterTriggerRestoreHook<?>> masterHooks, CheckpointIDCounter checkpointIDCounter, CompletedCheckpointStore checkpointStore, StateBackend checkpointStateBackend, CheckpointStorage checkpointStorage, CheckpointStatsTracker statsTracker, CheckpointsCleaner checkpointsCleaner, String changelogStorageName)
- Specified by:
enableCheckpointingin interfaceExecutionGraph
-
getCheckpointCoordinator
@Nullable public CheckpointCoordinator getCheckpointCoordinator()
- Specified by:
getCheckpointCoordinatorin interfaceExecutionGraph
-
getKvStateLocationRegistry
public KvStateLocationRegistry getKvStateLocationRegistry()
- Specified by:
getKvStateLocationRegistryin interfaceExecutionGraph
-
getCheckpointCoordinatorConfiguration
public CheckpointCoordinatorConfiguration getCheckpointCoordinatorConfiguration()
Description copied from interface:AccessExecutionGraphReturns theCheckpointCoordinatorConfigurationornullif checkpointing is disabled.- Specified by:
getCheckpointCoordinatorConfigurationin interfaceAccessExecutionGraph- Returns:
- JobCheckpointingConfiguration for this execution graph
-
getCheckpointStatsSnapshot
public CheckpointStatsSnapshot getCheckpointStatsSnapshot()
Description copied from interface:AccessExecutionGraphReturns a snapshot of the checkpoint statistics ornullif checkpointing is disabled.- Specified by:
getCheckpointStatsSnapshotin interfaceAccessExecutionGraph- Returns:
- Snapshot of the checkpoint statistics for this execution graph
-
setPlan
public void setPlan(JobPlanInfo.Plan plan)
- Specified by:
setPlanin interfaceExecutionGraph
-
getStreamGraphJson
public String getStreamGraphJson()
Description copied from interface:AccessExecutionGraphReturns the stream graph as a JSON string.- Specified by:
getStreamGraphJsonin interfaceAccessExecutionGraph- Returns:
- stream graph as a JSON string, or null if the job is submitted with a JobGraph or if it's a streaming job.
-
getPlan
public JobPlanInfo.Plan getPlan()
Description copied from interface:AccessExecutionGraphReturns the job plan as a JobPlanInfo.Plan.- Specified by:
getPlanin interfaceAccessExecutionGraph- Returns:
- job plan as a JobPlanInfo.Plan
-
getJobID
public org.apache.flink.api.common.JobID getJobID()
Description copied from interface:AccessExecutionGraphReturns theJobIDfor this execution graph.- Specified by:
getJobIDin interfaceAccessExecutionGraph- Specified by:
getJobIDin interfaceInternalExecutionGraphAccessor- Returns:
- job ID for this execution graph
-
getJobName
public String getJobName()
Description copied from interface:AccessExecutionGraphReturns the job name for the execution graph.- Specified by:
getJobNamein interfaceAccessExecutionGraph- Returns:
- job name for this execution graph
-
isStoppable
public boolean isStoppable()
Description copied from interface:AccessExecutionGraphReturns whether the job for this execution graph is stoppable.- Specified by:
isStoppablein interfaceAccessExecutionGraph- Returns:
- true, if all sources tasks are stoppable, false otherwise
-
getJobConfiguration
public org.apache.flink.configuration.Configuration getJobConfiguration()
- Specified by:
getJobConfigurationin interfaceExecutionGraph
-
getUserClassLoader
public ClassLoader getUserClassLoader()
- Specified by:
getUserClassLoaderin interfaceInternalExecutionGraphAccessor
-
getState
public org.apache.flink.api.common.JobStatus getState()
Description copied from interface:AccessExecutionGraphReturns the currentJobStatusfor this execution graph.- Specified by:
getStatein interfaceAccessExecutionGraph- Specified by:
getStatein interfaceJobStatusProvider- Returns:
- job status for this execution graph
-
getJobType
public JobType getJobType()
Description copied from interface:AccessExecutionGraphReturns theJobTypefor this execution graph.- Specified by:
getJobTypein interfaceAccessExecutionGraph- Returns:
- job type for this execution graph. It may be null when an exception occurs.
-
getFailureCause
public Throwable getFailureCause()
- Specified by:
getFailureCausein interfaceExecutionGraph
-
getFailureInfo
public ErrorInfo getFailureInfo()
Description copied from interface:AccessExecutionGraphReturns the exception that caused the job to fail. This is the first root exception that was not recoverable and triggered job failure.- Specified by:
getFailureInfoin interfaceAccessExecutionGraph- Returns:
- failure causing exception, or null
-
getNumberOfRestarts
public long getNumberOfRestarts()
Description copied from interface:ExecutionGraphGets the number of restarts, including full restarts and fine grained restarts. If a recovery is currently pending, this recovery is included in the count.- Specified by:
getNumberOfRestartsin interfaceExecutionGraph- Returns:
- The number of restarts so far
-
getNumFinishedVertices
public int getNumFinishedVertices()
- Specified by:
getNumFinishedVerticesin interfaceExecutionGraph
-
getJobVertex
public ExecutionJobVertex getJobVertex(JobVertexID id)
Description copied from interface:AccessExecutionGraphReturns the job vertex for the givenJobVertexID.- Specified by:
getJobVertexin interfaceAccessExecutionGraph- Specified by:
getJobVertexin interfaceExecutionGraph- Specified by:
getJobVertexin interfaceInternalExecutionGraphAccessor- Parameters:
id- id of job vertex to be returned- Returns:
- job vertex for the given id, or
null
-
getAllVertices
public Map<JobVertexID,ExecutionJobVertex> getAllVertices()
Description copied from interface:AccessExecutionGraphReturns a map containing all job vertices for this execution graph.- Specified by:
getAllVerticesin interfaceAccessExecutionGraph- Specified by:
getAllVerticesin interfaceExecutionGraph- Returns:
- map containing all job vertices for this execution graph
-
getVerticesTopologically
public Iterable<ExecutionJobVertex> getVerticesTopologically()
Description copied from interface:AccessExecutionGraphReturns an iterable containing all job vertices for this execution graph in the order they were created.- Specified by:
getVerticesTopologicallyin interfaceAccessExecutionGraph- Specified by:
getVerticesTopologicallyin interfaceExecutionGraph- Returns:
- iterable containing all job vertices for this execution graph in the order they were created
-
getAllIntermediateResults
public Map<IntermediateDataSetID,IntermediateResult> getAllIntermediateResults()
- Specified by:
getAllIntermediateResultsin interfaceExecutionGraph
-
getAllExecutionVertices
public Iterable<ExecutionVertex> getAllExecutionVertices()
Description copied from interface:AccessExecutionGraphReturns an iterable containing all execution vertices for this execution graph.- Specified by:
getAllExecutionVerticesin interfaceAccessExecutionGraph- Specified by:
getAllExecutionVerticesin interfaceExecutionGraph- Returns:
- iterable containing all execution vertices for this execution graph
-
getEdgeManager
public EdgeManager getEdgeManager()
- Specified by:
getEdgeManagerin interfaceInternalExecutionGraphAccessor
-
getExecutionVertexOrThrow
public ExecutionVertex getExecutionVertexOrThrow(ExecutionVertexID id)
- Specified by:
getExecutionVertexOrThrowin interfaceInternalExecutionGraphAccessor
-
getResultPartitionOrThrow
public IntermediateResultPartition getResultPartitionOrThrow(IntermediateResultPartitionID id)
Description copied from interface:ExecutionGraphGets the intermediate result partition by the given partition ID, or throw an exception if the partition is not found.- Specified by:
getResultPartitionOrThrowin interfaceExecutionGraph- Specified by:
getResultPartitionOrThrowin interfaceInternalExecutionGraphAccessor- Parameters:
id- of the intermediate result partition- Returns:
- intermediate result partition
-
getStatusTimestamp
public long getStatusTimestamp(org.apache.flink.api.common.JobStatus status)
Description copied from interface:AccessExecutionGraphReturns the timestamp for the givenJobStatus.- Specified by:
getStatusTimestampin interfaceAccessExecutionGraph- Specified by:
getStatusTimestampin interfaceJobStatusProvider- Parameters:
status- status for which the timestamp should be returned- Returns:
- timestamp for the given job status
-
getBlobWriter
public final BlobWriter getBlobWriter()
- Specified by:
getBlobWriterin interfaceInternalExecutionGraphAccessor
-
getFutureExecutor
public Executor getFutureExecutor()
Description copied from interface:InternalExecutionGraphAccessorReturns the ExecutionContext associated with this ExecutionGraph.- Specified by:
getFutureExecutorin interfaceInternalExecutionGraphAccessor- Returns:
- ExecutionContext associated with this ExecutionGraph
-
aggregateUserAccumulators
public Map<String,org.apache.flink.util.OptionalFailure<org.apache.flink.api.common.accumulators.Accumulator<?,?>>> aggregateUserAccumulators()
Description copied from interface:ExecutionGraphMerges all accumulator results from the tasks previously executed in the Executions.- Specified by:
aggregateUserAccumulatorsin interfaceExecutionGraph- Returns:
- The accumulator map
-
getAccumulatorsSerialized
public Map<String,org.apache.flink.util.SerializedValue<org.apache.flink.util.OptionalFailure<Object>>> getAccumulatorsSerialized()
Gets a serialized accumulator map.- Specified by:
getAccumulatorsSerializedin interfaceAccessExecutionGraph- Returns:
- The accumulator map with serialized accumulator values.
-
getAccumulatorResultsStringified
public StringifiedAccumulatorResult[] getAccumulatorResultsStringified()
Returns the a stringified version of the user-defined accumulators.- Specified by:
getAccumulatorResultsStringifiedin interfaceAccessExecutionGraph- Returns:
- an Array containing the StringifiedAccumulatorResult objects
-
setInternalTaskFailuresListener
public void setInternalTaskFailuresListener(InternalFailuresListener internalTaskFailuresListener)
- Specified by:
setInternalTaskFailuresListenerin interfaceExecutionGraph
-
notifyNewlyInitializedJobVertices
public void notifyNewlyInitializedJobVertices(List<ExecutionJobVertex> vertices)
Description copied from interface:ExecutionGraphNotify that some job vertices have been newly initialized, execution graph will try to update scheduling topology.- Specified by:
notifyNewlyInitializedJobVerticesin interfaceExecutionGraph- Parameters:
vertices- The execution job vertices that are newly initialized.
-
attachJobGraph
public void attachJobGraph(List<JobVertex> verticesToAttach, JobManagerJobMetricGroup jobManagerJobMetricGroup) throws JobException
- Specified by:
attachJobGraphin interfaceExecutionGraph- Throws:
JobException
-
addNewJobVertices
public void addNewJobVertices(List<JobVertex> topologicallySortedNewlyJobVertices, JobManagerJobMetricGroup jobManagerJobMetricGroup, VertexParallelismStore newVerticesParallelismStore) throws JobException
Description copied from interface:ExecutionGraphAdds new job vertices to the execution graph based on the provided list of topologically sorted job vertices.- Specified by:
addNewJobVerticesin interfaceExecutionGraph- Parameters:
topologicallySortedNewlyJobVertices- a list of job vertices that are to be added, defined in topological order.jobManagerJobMetricGroup- the metric group associated with the job manager for monitoring and metrics collection.newVerticesParallelismStore- a store that maintains parallelism information for the newly added job vertices.- Throws:
JobException
-
initializeJobVertex
public void initializeJobVertex(ExecutionJobVertex ejv, long createTimestamp, Map<IntermediateDataSetID,JobVertexInputInfo> jobVertexInputInfos) throws JobException
Description copied from interface:ExecutionGraphInitialize the given execution job vertex, mainly includes creating execution vertices according to the parallelism, and connecting to the predecessors.- Specified by:
initializeJobVertexin interfaceExecutionGraph- Parameters:
ejv- The execution job vertex that needs to be initialized.createTimestamp- The timestamp for creating execution vertices, used to initialize the first Execution with.jobVertexInputInfos- The input infos of this job vertex.- Throws:
JobException
-
transitionToRunning
public void transitionToRunning()
- Specified by:
transitionToRunningin interfaceExecutionGraph
-
cancel
public void cancel()
- Specified by:
cancelin interfaceExecutionGraph
-
suspend
public void suspend(Throwable suspensionCause)
Description copied from interface:ExecutionGraphSuspends the current ExecutionGraph.The JobStatus will be directly set to
JobStatus.SUSPENDEDiff the current state is not a terminal state. All ExecutionJobVertices will be canceled and the onTerminalState() is executed.The
JobStatus.SUSPENDEDstate is a local terminal state which stops the execution of the job but does not remove the job from the HA job store so that it can be recovered by another JobManager.- Specified by:
suspendin interfaceExecutionGraph- Parameters:
suspensionCause- Cause of the suspension
-
failGlobal
public void failGlobal(Throwable t)
Description copied from interface:InternalExecutionGraphAccessorFails the execution graph globally.This global failure is meant to be triggered in cases where the consistency of the execution graph' state cannot be guaranteed any more (for example when catching unexpected exceptions that indicate a bug or an unexpected call race), and where a full restart is the safe way to get consistency back.
- Specified by:
failGlobalin interfaceInternalExecutionGraphAccessor- Parameters:
t- The exception that caused the failure.
-
getArchivedExecutionConfig
public org.apache.flink.api.common.ArchivedExecutionConfig getArchivedExecutionConfig()
Returns the serializableArchivedExecutionConfig.- Specified by:
getArchivedExecutionConfigin interfaceAccessExecutionGraph- Returns:
- ArchivedExecutionConfig which may be null in case of errors
-
getTerminationFuture
public CompletableFuture<org.apache.flink.api.common.JobStatus> getTerminationFuture()
Description copied from interface:ExecutionGraphReturns the termination future of thisExecutionGraph. The termination future is completed with the terminalJobStatusonce the ExecutionGraph reaches this terminal state and allExecutionhave been terminated.- Specified by:
getTerminationFuturein interfaceExecutionGraph- Returns:
- Termination future of this
ExecutionGraph.
-
waitUntilTerminal
@VisibleForTesting public org.apache.flink.api.common.JobStatus waitUntilTerminal() throws InterruptedException- Specified by:
waitUntilTerminalin interfaceExecutionGraph- Throws:
InterruptedException
-
transitionState
public boolean transitionState(org.apache.flink.api.common.JobStatus current, org.apache.flink.api.common.JobStatus newState)- Specified by:
transitionStatein interfaceExecutionGraph
-
incrementRestarts
public void incrementRestarts()
- Specified by:
incrementRestartsin interfaceExecutionGraph
-
initFailureCause
public void initFailureCause(Throwable t, long timestamp)
- Specified by:
initFailureCausein interfaceExecutionGraph
-
jobVertexFinished
public void jobVertexFinished()
Called whenever a job vertex reaches state FINISHED (completed successfully). Once all job vertices are in the FINISHED state, the program is successfully done.- Specified by:
jobVertexFinishedin interfaceInternalExecutionGraphAccessor
-
jobVertexUnFinished
public void jobVertexUnFinished()
- Specified by:
jobVertexUnFinishedin interfaceInternalExecutionGraphAccessor
-
failJob
public void failJob(Throwable cause, long timestamp)
- Specified by:
failJobin interfaceExecutionGraph
-
updateState
public boolean updateState(TaskExecutionStateTransition state)
Description copied from interface:ExecutionGraphUpdates the state of one of the ExecutionVertex's Execution attempts. If the new status if "FINISHED", this also updates the accumulators.- Specified by:
updateStatein interfaceExecutionGraph- Parameters:
state- The state update.- Returns:
- True, if the task update was properly applied, false, if the execution attempt was not found.
-
createResultPartitionId
@VisibleForTesting public ResultPartitionID createResultPartitionId(IntermediateResultPartitionID resultPartitionId)
-
getRegisteredExecutions
public Map<ExecutionAttemptID,Execution> getRegisteredExecutions()
- Specified by:
getRegisteredExecutionsin interfaceExecutionGraph
-
registerExecution
public void registerExecution(Execution exec)
- Specified by:
registerExecutionin interfaceInternalExecutionGraphAccessor
-
deregisterExecution
public void deregisterExecution(Execution exec)
- Specified by:
deregisterExecutionin interfaceInternalExecutionGraphAccessor
-
updateAccumulators
public void updateAccumulators(AccumulatorSnapshot accumulatorSnapshot)
Description copied from interface:ExecutionGraphUpdates the accumulators during the runtime of a job. Final accumulator results are transferred through the UpdateTaskExecutionState message.- Specified by:
updateAccumulatorsin interfaceExecutionGraph- Parameters:
accumulatorSnapshot- The serialized flink and user-defined accumulators
-
registerJobStatusListener
public void registerJobStatusListener(JobStatusListener listener)
- Specified by:
registerJobStatusListenerin interfaceExecutionGraph
-
notifyExecutionChange
public void notifyExecutionChange(Execution execution, ExecutionState previousState, ExecutionState newExecutionState)
- Specified by:
notifyExecutionChangein interfaceInternalExecutionGraphAccessor
-
notifySchedulerNgAboutInternalTaskFailure
public void notifySchedulerNgAboutInternalTaskFailure(ExecutionAttemptID attemptId, Throwable t, boolean cancelTask, boolean releasePartitions)
- Specified by:
notifySchedulerNgAboutInternalTaskFailurein interfaceInternalExecutionGraphAccessor
-
deleteBlobs
public void deleteBlobs(List<PermanentBlobKey> blobKeys)
- Specified by:
deleteBlobsin interfaceInternalExecutionGraphAccessor
-
getShuffleMaster
public ShuffleMaster<?> getShuffleMaster()
- Specified by:
getShuffleMasterin interfaceInternalExecutionGraphAccessor
-
getPartitionTracker
public JobMasterPartitionTracker getPartitionTracker()
- Specified by:
getPartitionTrackerin interfaceInternalExecutionGraphAccessor
-
getResultPartitionAvailabilityChecker
public ResultPartitionAvailabilityChecker getResultPartitionAvailabilityChecker()
- Specified by:
getResultPartitionAvailabilityCheckerin interfaceExecutionGraph
-
getPartitionGroupReleaseStrategy
public PartitionGroupReleaseStrategy getPartitionGroupReleaseStrategy()
- Specified by:
getPartitionGroupReleaseStrategyin interfaceInternalExecutionGraphAccessor
-
getExecutionDeploymentListener
public ExecutionDeploymentListener getExecutionDeploymentListener()
- Specified by:
getExecutionDeploymentListenerin interfaceInternalExecutionGraphAccessor
-
isDynamic
public boolean isDynamic()
- Specified by:
isDynamicin interfaceInternalExecutionGraphAccessor
-
findVertexWithAttempt
public Optional<String> findVertexWithAttempt(ExecutionAttemptID attemptId)
- Specified by:
findVertexWithAttemptin interfaceExecutionGraph
-
findExecution
public Optional<AccessExecution> findExecution(ExecutionAttemptID attemptId)
- Specified by:
findExecutionin interfaceExecutionGraph
-
getExecutionGraphID
public ExecutionGraphID getExecutionGraphID()
- Specified by:
getExecutionGraphIDin interfaceInternalExecutionGraphAccessor
-
getClusterPartitionShuffleDescriptors
public List<ShuffleDescriptor> getClusterPartitionShuffleDescriptors(IntermediateDataSetID intermediateDataSetID)
Description copied from interface:InternalExecutionGraphAccessorGet the shuffle descriptors of the cluster partitions ordered by partition number.- Specified by:
getClusterPartitionShuffleDescriptorsin interfaceInternalExecutionGraphAccessor
-
getMarkPartitionFinishedStrategy
public MarkPartitionFinishedStrategy getMarkPartitionFinishedStrategy()
- Specified by:
getMarkPartitionFinishedStrategyin interfaceInternalExecutionGraphAccessor
-
getJobVertexInputInfo
public JobVertexInputInfo getJobVertexInputInfo(JobVertexID jobVertexId, IntermediateDataSetID resultId)
Description copied from interface:InternalExecutionGraphAccessorGet the input info of a certain input of a certain job vertex.- Specified by:
getJobVertexInputInfoin interfaceInternalExecutionGraphAccessor- Parameters:
jobVertexId- the job vertex idresultId- the input(intermediate result) id- Returns:
- the input info
-
getTaskDeploymentDescriptorFactory
public TaskDeploymentDescriptorFactory getTaskDeploymentDescriptorFactory()
- Specified by:
getTaskDeploymentDescriptorFactoryin interfaceInternalExecutionGraphAccessor
-
-