com.google.cloud.dataflow.sdk.options

Interface DataflowPipelineWorkerPoolOptions

All Superinterfaces:

PipelineOptions

All Known Subinterfaces:

BlockingDataflowPipelineOptions, DataflowPipelineOptions, DataflowWorkerHarnessOptions, TestDataflowPipelineOptions

public interface DataflowPipelineWorkerPoolOptions
extends PipelineOptions

Options that are used to configure the Dataflow pipeline worker pool.

Nested Class Summary

Nested Classes

Modifier and Type	Interface and Description
static class	DataflowPipelineWorkerPoolOptions.AutoscalingAlgorithmType Type of autoscaling algorithm to use.
static class	DataflowPipelineWorkerPoolOptions.ClusterManagerApiType Type of API for handling cluster management, i.e.
static class	DataflowPipelineWorkerPoolOptions.TeardownPolicy The policy for tearing down the workers spun up by the service.

Nested classes/interfaces inherited from interface com.google.cloud.dataflow.sdk.options.PipelineOptions

PipelineOptions.CheckEnabled

Method Summary

Modifier and Type	Method and Description
DataflowPipelineWorkerPoolOptions.AutoscalingAlgorithmType	getAutoscalingAlgorithm()
DataflowPipelineWorkerPoolOptions.ClusterManagerApiType	getClusterManagerApi() Type of API for handling cluster management, i.e.
int	getDiskSizeGb() Remote worker disk size, in gigabytes, or 0 to use the default size.
String	getDiskSourceImage() Disk source image to use by VMs for jobs.
List<String>	getFilesToStage() List of local files to make available to workers.
int	getMaxNumWorkers() The maximum number of workers to use when using workerpool autoscaling.
String	getNetwork() GCE network for launching workers.
int	getNumWorkers() Number of workers to use when executing the Dataflow job.
DataflowPipelineWorkerPoolOptions.TeardownPolicy	getTeardownPolicy() The teardown policy for the VMs.
String	getWorkerDiskType() Specifies what type of persistent disk should be used.
String	getWorkerMachineType() Machine type to create Dataflow worker VMs as.
String	getWorkerPoolType() Specifies what type of worker pool should be used.
String	getZone() GCE availability zone for launching workers.
void	setAutoscalingAlgorithm(DataflowPipelineWorkerPoolOptions.AutoscalingAlgorithmType value)
void	setClusterManagerApi(DataflowPipelineWorkerPoolOptions.ClusterManagerApiType value)
void	setDiskSizeGb(int value)
void	setDiskSourceImage(String value)
void	setFilesToStage(List<String> value)
void	setMaxNumWorkers(int value)
void	setNetwork(String value)
void	setNumWorkers(int value)
void	setTeardownPolicy(DataflowPipelineWorkerPoolOptions.TeardownPolicy value)
void	setWorkerDiskType(String value)
void	setWorkerMachineType(String value)
void	setWorkerPoolType(String value)
void	setZone(String value)

Methods inherited from interface com.google.cloud.dataflow.sdk.options.PipelineOptions

as, cloneAs, getRunner, getStableUniqueNames, setRunner, setStableUniqueNames

Method Detail

getDiskSourceImage

String getDiskSourceImage()

Disk source image to use by VMs for jobs.

See Also:

Compute Engine Images

setDiskSourceImage

void setDiskSourceImage(String value)

getNumWorkers

int getNumWorkers()

Number of workers to use when executing the Dataflow job.

setNumWorkers

void setNumWorkers(int value)

getAutoscalingAlgorithm

@Experimental(value=AUTOSCALING)
DataflowPipelineWorkerPoolOptions.AutoscalingAlgorithmType getAutoscalingAlgorithm()

setAutoscalingAlgorithm

void setAutoscalingAlgorithm(DataflowPipelineWorkerPoolOptions.AutoscalingAlgorithmType value)

getMaxNumWorkers

@Experimental(value=AUTOSCALING)
int getMaxNumWorkers()

The maximum number of workers to use when using workerpool autoscaling.

setMaxNumWorkers

void setMaxNumWorkers(int value)

getDiskSizeGb

int getDiskSizeGb()

Remote worker disk size, in gigabytes, or 0 to use the default size.

setDiskSizeGb

void setDiskSizeGb(int value)

getNetwork

String getNetwork()

GCE network for launching workers.

Default is up to the Dataflow service.

setNetwork

void setNetwork(String value)

getZone

String getZone()

GCE availability zone for launching workers.

Default is up to the Dataflow service.

setZone

void setZone(String value)

getClusterManagerApi

DataflowPipelineWorkerPoolOptions.ClusterManagerApiType getClusterManagerApi()

Type of API for handling cluster management, i.e. resizing, healthchecking, etc.

setClusterManagerApi

void setClusterManagerApi(DataflowPipelineWorkerPoolOptions.ClusterManagerApiType value)

getWorkerMachineType

String getWorkerMachineType()

Machine type to create Dataflow worker VMs as.

See GCE machine types for a list of valid options.

If unset, the Dataflow service will choose a reasonable default.

setWorkerMachineType

void setWorkerMachineType(String value)

getTeardownPolicy

DataflowPipelineWorkerPoolOptions.TeardownPolicy getTeardownPolicy()

The teardown policy for the VMs.

By default this is left unset and the service sets the default policy.

setTeardownPolicy

void setTeardownPolicy(DataflowPipelineWorkerPoolOptions.TeardownPolicy value)

getFilesToStage

List<String> getFilesToStage()

List of local files to make available to workers.

Files are placed on the worker's classpath.

The default value is the list of jars from the main program's classpath.

setFilesToStage

void setFilesToStage(List<String> value)

getWorkerPoolType

String getWorkerPoolType()

Specifies what type of worker pool should be used.

This is an internal option, and should not be set when using the Dataflow service.

setWorkerPoolType

void setWorkerPoolType(String value)

getWorkerDiskType

String getWorkerDiskType()

Specifies what type of persistent disk should be used.

setWorkerDiskType

void setWorkerDiskType(String value)