com.google.cloud.dataflow.sdk.options

Interface DataflowPipelineOptions

All Superinterfaces:

ApplicationNameOptions, BigQueryOptions, CloudDebuggerOptions, DataflowPipelineDebugOptions, DataflowPipelineWorkerPoolOptions, DataflowWorkerLoggingOptions, GcpOptions, GcsOptions, GoogleApiDebugOptions, PipelineOptions, StreamingOptions

All Known Subinterfaces:

BlockingDataflowPipelineOptions, DataflowWorkerHarnessOptions, TestDataflowPipelineOptions

public interface DataflowPipelineOptions
extends PipelineOptions, GcpOptions, ApplicationNameOptions, DataflowPipelineDebugOptions, DataflowPipelineWorkerPoolOptions, BigQueryOptions, GcsOptions, StreamingOptions, CloudDebuggerOptions, DataflowWorkerLoggingOptions

Options that can be used to configure the DataflowPipeline.

Nested Class Summary

Nested Classes

Modifier and Type	Interface and Description
static class	DataflowPipelineOptions.JobNameFactory Returns a normalized job name constructed from ApplicationNameOptions.getAppName(), the local system user name (if available), and the current time.

Nested classes/interfaces inherited from interface com.google.cloud.dataflow.sdk.options.DataflowPipelineDebugOptions

DataflowPipelineDebugOptions.DataflowClientFactory, DataflowPipelineDebugOptions.PathValidatorFactory, DataflowPipelineDebugOptions.StagerFactory

Nested classes/interfaces inherited from interface com.google.cloud.dataflow.sdk.options.DataflowPipelineWorkerPoolOptions

DataflowPipelineWorkerPoolOptions.AutoscalingAlgorithmType, DataflowPipelineWorkerPoolOptions.TeardownPolicy

Nested classes/interfaces inherited from interface com.google.cloud.dataflow.sdk.options.GcpOptions

GcpOptions.CredentialDirFactory, GcpOptions.DefaultProjectFactory, GcpOptions.GcpUserCredentialsFactory

Nested classes/interfaces inherited from interface com.google.cloud.dataflow.sdk.options.GoogleApiDebugOptions

GoogleApiDebugOptions.GoogleApiTracer

Nested classes/interfaces inherited from interface com.google.cloud.dataflow.sdk.options.PipelineOptions

PipelineOptions.CheckEnabled

Nested classes/interfaces inherited from interface com.google.cloud.dataflow.sdk.options.GcsOptions

GcsOptions.ExecutorServiceFactory

Nested classes/interfaces inherited from interface com.google.cloud.dataflow.sdk.options.DataflowWorkerLoggingOptions

DataflowWorkerLoggingOptions.Level, DataflowWorkerLoggingOptions.WorkerLogLevelOverrides

Field Summary

Fields

Modifier and Type	Field and Description
static String	DATAFLOW_STORAGE_LOCATION

Method Summary

Modifier and Type	Method and Description
String	getJobName() The Dataflow job name is used as an idempotence key within the Dataflow service.
String	getProject() Project id to use when launching jobs.
String	getStagingLocation() GCS path for staging local files, e.g.
String	getTempLocation() GCS path for temporary files, e.g.
void	setJobName(String value)
void	setProject(String value)
void	setStagingLocation(String value)
void	setTempLocation(String value)

Methods inherited from interface com.google.cloud.dataflow.sdk.options.DataflowPipelineDebugOptions

getApiRootUrl, getDataflowClient, getDataflowEndpoint, getDataflowJobFile, getExperiments, getNumberOfWorkerHarnessThreads, getOverrideWindmillBinary, getPathValidator, getPathValidatorClass, getPubsubRootUrl, getStager, getStagerClass, getTransformNameMapping, getUpdate, setApiRootUrl, setDataflowClient, setDataflowEndpoint, setDataflowJobFile, setExperiments, setNumberOfWorkerHarnessThreads, setOverrideWindmillBinary, setPathValidator, setPathValidatorClass, setPubsubRootUrl, setStager, setStagerClass, setTransformNameMapping, setUpdate

Methods inherited from interface com.google.cloud.dataflow.sdk.options.DataflowPipelineWorkerPoolOptions

getAutoscalingAlgorithm, getDiskSizeGb, getDiskSourceImage, getFilesToStage, getMaxNumWorkers, getNetwork, getNumWorkers, getTeardownPolicy, getWorkerDiskType, getWorkerMachineType, getZone, setAutoscalingAlgorithm, setDiskSizeGb, setDiskSourceImage, setFilesToStage, setMaxNumWorkers, setNetwork, setNumWorkers, setTeardownPolicy, setWorkerDiskType, setWorkerMachineType, setZone

Methods inherited from interface com.google.cloud.dataflow.sdk.options.BigQueryOptions

getTempDatasetId, setTempDatasetId

Methods inherited from interface com.google.cloud.dataflow.sdk.options.StreamingOptions

isStreaming, setStreaming

Methods inherited from interface com.google.cloud.dataflow.sdk.options.ApplicationNameOptions

getAppName, setAppName

Methods inherited from interface com.google.cloud.dataflow.sdk.options.GcpOptions

getCredentialDir, getCredentialFactoryClass, getCredentialId, getGcpCredential, getSecretsFile, getServiceAccountKeyfile, getServiceAccountName, setCredentialDir, setCredentialFactoryClass, setCredentialId, setGcpCredential, setSecretsFile, setServiceAccountKeyfile, setServiceAccountName

Methods inherited from interface com.google.cloud.dataflow.sdk.options.GoogleApiDebugOptions

getGoogleApiTrace, setGoogleApiTrace

Methods inherited from interface com.google.cloud.dataflow.sdk.options.PipelineOptions

as, cloneAs, getRunner, getStableUniqueNames, setRunner, setStableUniqueNames

Methods inherited from interface com.google.cloud.dataflow.sdk.options.GcsOptions

getExecutorService, getGcsEndpoint, getGcsUtil, setExecutorService, setGcsEndpoint, setGcsUtil

Methods inherited from interface com.google.cloud.dataflow.sdk.options.CloudDebuggerOptions

getEnableCloudDebugger, setEnableCloudDebugger

Methods inherited from interface com.google.cloud.dataflow.sdk.options.DataflowWorkerLoggingOptions

getDefaultWorkerLogLevel, getWorkerLogLevelOverrides, setDefaultWorkerLogLevel, setWorkerLogLevelOverrides

Field Detail

DATAFLOW_STORAGE_LOCATION

static final String DATAFLOW_STORAGE_LOCATION

See Also:

Constant Field Values

Method Detail

getProject

String getProject()

Description copied from interface: GcpOptions

Project id to use when launching jobs.

Specified by:

getProject in interface GcpOptions

setProject

void setProject(String value)

Specified by:

setProject in interface GcpOptions

getTempLocation

String getTempLocation()

GCS path for temporary files, e.g. gs://bucket/object

Must be a valid Cloud Storage url, beginning with the prefix "gs://"

At least one of getTempLocation() or getStagingLocation() must be set. If getTempLocation() is not set, then the Dataflow pipeline defaults to using getStagingLocation().

setTempLocation

void setTempLocation(String value)

getStagingLocation

String getStagingLocation()

GCS path for staging local files, e.g. gs://bucket/object

Must be a valid Cloud Storage url, beginning with the prefix "gs://"

At least one of getTempLocation() or getStagingLocation() must be set. If getTempLocation() is not set, then the Dataflow pipeline defaults to using getStagingLocation().

setStagingLocation

void setStagingLocation(String value)

getJobName

String getJobName()

The Dataflow job name is used as an idempotence key within the Dataflow service. If there is an existing job that is currently active, another active job with the same name will not be able to be created. Defaults to using the ApplicationName-UserName-Date.

setJobName

void setJobName(String value)