org.apache.flink.runtime.minicluster

Class MiniCluster

java.lang.Object

org.apache.flink.runtime.minicluster.MiniCluster

All Implemented Interfaces:

AutoCloseable, JobExecutor, JobExecutorService, org.apache.flink.util.AutoCloseableAsync

public class MiniCluster
extends Object
implements JobExecutorService, org.apache.flink.util.AutoCloseableAsync

MiniCluster to execute Flink jobs locally.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected static class	MiniCluster.CommonRpcServiceFactory Factory which returns always the common RpcService.
protected class	MiniCluster.DedicatedRpcServiceFactory Factory which creates and registers new RpcService.
protected static interface	MiniCluster.RpcServiceFactory Internal factory for RpcService.

Constructor Summary

Constructors

Constructor and Description
MiniCluster(MiniClusterConfiguration miniClusterConfiguration) Creates a new Flink mini cluster based on the given configuration.

Method Summary

Modifier and Type	Method and Description
CompletableFuture<Acknowledge>	cancelJob(org.apache.flink.api.common.JobID jobId)
CompletableFuture<Void>	closeAsync() Shuts down the mini cluster, failing all currently executing jobs.
protected Collection<? extends DispatcherResourceManagerComponent>	createDispatcherResourceManagerComponents(org.apache.flink.configuration.Configuration configuration, MiniCluster.RpcServiceFactory rpcServiceFactory, HighAvailabilityServices haServices, BlobServer blobServer, HeartbeatServices heartbeatServices, MetricRegistry metricRegistry, MetricQueryServiceRetriever metricQueryServiceRetriever, FatalErrorHandler fatalErrorHandler)
protected HighAvailabilityServices	createHighAvailabilityServices(org.apache.flink.configuration.Configuration configuration, Executor executor)
protected RpcService	createLocalRpcService(org.apache.flink.configuration.Configuration configuration) Factory method to instantiate the local RPC service.
protected MetricRegistryImpl	createMetricRegistry(org.apache.flink.configuration.Configuration config) Factory method to create the metric registry for the mini cluster.
protected RpcService	createRemoteRpcService(org.apache.flink.configuration.Configuration configuration, String bindAddress, int bindPort) Factory method to instantiate the remote RPC service.
protected RpcService	createRemoteRpcService(org.apache.flink.configuration.Configuration configuration, String externalAddress, String externalPortRange, String bindAddress) Factory method to instantiate the remote RPC service.
CompletableFuture<CoordinationResponse>	deliverCoordinationRequestToCoordinator(org.apache.flink.api.common.JobID jobId, OperatorID operatorId, org.apache.flink.util.SerializedValue<CoordinationRequest> serializedRequest)
CompletableFuture<Acknowledge>	disposeSavepoint(String savepointPath)
org.apache.flink.api.common.JobExecutionResult	executeJobBlocking(JobGraph job) This method runs a job in blocking mode.
ClusterInformation	getClusterInformation()
protected CompletableFuture<DispatcherGateway>	getDispatcherGatewayFuture()
CompletableFuture<? extends AccessExecutionGraph>	getExecutionGraph(org.apache.flink.api.common.JobID jobId)
protected Executor	getIOExecutor()
CompletableFuture<org.apache.flink.api.common.JobStatus>	getJobStatus(org.apache.flink.api.common.JobID jobId)
CompletableFuture<URI>	getRestAddress()
boolean	isRunning() Checks if the mini cluster was started and is running.
CompletableFuture<Collection<JobStatusMessage>>	listJobs()
CompletableFuture<ClusterOverview>	requestClusterOverview()
CompletableFuture<JobResult>	requestJobResult(org.apache.flink.api.common.JobID jobId)
void	runDetached(JobGraph job) This method executes a job in detached mode.
void	start() Starts the mini cluster, based on the configured properties.
CompletableFuture<String>	stopWithSavepoint(org.apache.flink.api.common.JobID jobId, String targetDirectory, boolean advanceToEndOfEventTime)
CompletableFuture<org.apache.flink.api.common.JobSubmissionResult>	submitJob(JobGraph jobGraph)
protected CompletableFuture<Void>	terminateTaskExecutor(int index)
CompletableFuture<String>	triggerSavepoint(org.apache.flink.api.common.JobID jobId, String targetDirectory, boolean cancelJob)
protected boolean	useLocalCommunication()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.util.AutoCloseableAsync

close

Constructor Detail

MiniCluster

public MiniCluster(MiniClusterConfiguration miniClusterConfiguration)

Creates a new Flink mini cluster based on the given configuration.

Parameters:

miniClusterConfiguration - The configuration for the mini cluster

Method Detail

getRestAddress

public CompletableFuture<URI> getRestAddress()

getClusterInformation

public ClusterInformation getClusterInformation()

getIOExecutor

protected Executor getIOExecutor()

isRunning

public boolean isRunning()

Checks if the mini cluster was started and is running.

start

public void start()
           throws Exception

Starts the mini cluster, based on the configured properties.

Throws:

Exception - This method passes on any exception that occurs during the startup of the mini cluster.

createDispatcherResourceManagerComponents

@VisibleForTesting
protected Collection<? extends DispatcherResourceManagerComponent> createDispatcherResourceManagerComponents(org.apache.flink.configuration.Configuration configuration,
                                                                                                                                MiniCluster.RpcServiceFactory rpcServiceFactory,
                                                                                                                                HighAvailabilityServices haServices,
                                                                                                                                BlobServer blobServer,
                                                                                                                                HeartbeatServices heartbeatServices,
                                                                                                                                MetricRegistry metricRegistry,
                                                                                                                                MetricQueryServiceRetriever metricQueryServiceRetriever,
                                                                                                                                FatalErrorHandler fatalErrorHandler)
                                                                                                                         throws Exception

Throws:

Exception

createHighAvailabilityServices

@VisibleForTesting
protected HighAvailabilityServices createHighAvailabilityServices(org.apache.flink.configuration.Configuration configuration,
                                                                                     Executor executor)
                                                                              throws Exception

Throws:

Exception

closeAsync

public CompletableFuture<Void> closeAsync()

Shuts down the mini cluster, failing all currently executing jobs. The mini cluster can be started again by calling the start() method again.

This method shuts down all started services and components, even if an exception occurs in the process of shutting down some component.

Specified by:

closeAsync in interface org.apache.flink.util.AutoCloseableAsync

Returns:

Future which is completed once the MiniCluster has been completely shut down

useLocalCommunication

@VisibleForTesting
protected boolean useLocalCommunication()

terminateTaskExecutor

@VisibleForTesting
 @Nonnull
protected CompletableFuture<Void> terminateTaskExecutor(int index)

listJobs

public CompletableFuture<Collection<JobStatusMessage>> listJobs()

getJobStatus

public CompletableFuture<org.apache.flink.api.common.JobStatus> getJobStatus(org.apache.flink.api.common.JobID jobId)

cancelJob

public CompletableFuture<Acknowledge> cancelJob(org.apache.flink.api.common.JobID jobId)

triggerSavepoint

public CompletableFuture<String> triggerSavepoint(org.apache.flink.api.common.JobID jobId,
                                                  String targetDirectory,
                                                  boolean cancelJob)

stopWithSavepoint

public CompletableFuture<String> stopWithSavepoint(org.apache.flink.api.common.JobID jobId,
                                                   String targetDirectory,
                                                   boolean advanceToEndOfEventTime)

disposeSavepoint

public CompletableFuture<Acknowledge> disposeSavepoint(String savepointPath)

getExecutionGraph

public CompletableFuture<? extends AccessExecutionGraph> getExecutionGraph(org.apache.flink.api.common.JobID jobId)

deliverCoordinationRequestToCoordinator

public CompletableFuture<CoordinationResponse> deliverCoordinationRequestToCoordinator(org.apache.flink.api.common.JobID jobId,
                                                                                       OperatorID operatorId,
                                                                                       org.apache.flink.util.SerializedValue<CoordinationRequest> serializedRequest)

runDetached

public void runDetached(JobGraph job)
                 throws JobExecutionException,
                        InterruptedException

This method executes a job in detached mode. The method returns immediately after the job has been added to the

Parameters:

job - The Flink job to execute

Throws:

JobExecutionException - Thrown if anything went amiss during initial job launch, or if the job terminally failed.

InterruptedException

executeJobBlocking

public org.apache.flink.api.common.JobExecutionResult executeJobBlocking(JobGraph job)
                                                                  throws JobExecutionException,
                                                                         InterruptedException

This method runs a job in blocking mode. The method returns only after the job completed successfully, or after it failed terminally.

Specified by:

executeJobBlocking in interface JobExecutor

Parameters:

job - The Flink job to execute

Returns:

The result of the job execution

Throws:

JobExecutionException - Thrown if anything went amiss during initial job launch, or if the job terminally failed.

InterruptedException

submitJob

public CompletableFuture<org.apache.flink.api.common.JobSubmissionResult> submitJob(JobGraph jobGraph)

requestJobResult

public CompletableFuture<JobResult> requestJobResult(org.apache.flink.api.common.JobID jobId)

requestClusterOverview

public CompletableFuture<ClusterOverview> requestClusterOverview()

getDispatcherGatewayFuture

@VisibleForTesting
protected CompletableFuture<DispatcherGateway> getDispatcherGatewayFuture()

createMetricRegistry

protected MetricRegistryImpl createMetricRegistry(org.apache.flink.configuration.Configuration config)

Factory method to create the metric registry for the mini cluster.

Parameters:

config - The configuration of the mini cluster

createRemoteRpcService

protected RpcService createRemoteRpcService(org.apache.flink.configuration.Configuration configuration,
                                            String bindAddress,
                                            int bindPort)
                                     throws Exception

Factory method to instantiate the remote RPC service.

Parameters:

configuration - Flink configuration.

bindAddress - The address to bind the RPC service to.

bindPort - The port range to bind the RPC service to.

Returns:

The instantiated RPC service

Throws:

Exception

createRemoteRpcService

protected RpcService createRemoteRpcService(org.apache.flink.configuration.Configuration configuration,
                                            String externalAddress,
                                            String externalPortRange,
                                            String bindAddress)
                                     throws Exception

Factory method to instantiate the remote RPC service.

Parameters:

configuration - Flink configuration.

externalAddress - The external address to access the RPC service.

externalPortRange - The external port range to access the RPC service.

bindAddress - The address to bind the RPC service to.

Returns:

The instantiated RPC service

Throws:

Exception

createLocalRpcService

protected RpcService createLocalRpcService(org.apache.flink.configuration.Configuration configuration)
                                    throws Exception

Factory method to instantiate the local RPC service.

Parameters:

configuration - Flink configuration.

Returns:

The instantiated RPC service

Throws:

Exception