com.google.cloud.dataflow.sdk.options

Interface DataflowPipelineOptions

All Superinterfaces:

ApplicationNameOptions, BigQueryOptions, CloudDebuggerOptions, DataflowPipelineDebugOptions, DataflowPipelineWorkerPoolOptions, DataflowProfilingOptions, DataflowWorkerLoggingOptions, GcpOptions, GcsOptions, GoogleApiDebugOptions, HasDisplayData, PipelineOptions, StreamingOptions

All Known Subinterfaces:

BlockingDataflowPipelineOptions, DataflowWorkerHarnessOptions, TestDataflowPipelineOptions

public interface DataflowPipelineOptions
extends PipelineOptions, GcpOptions, ApplicationNameOptions, DataflowPipelineDebugOptions, DataflowPipelineWorkerPoolOptions, BigQueryOptions, GcsOptions, StreamingOptions, CloudDebuggerOptions, DataflowWorkerLoggingOptions, DataflowProfilingOptions

Options that can be used to configure the DataflowPipeline.

Nested Class Summary

Nested Classes

Modifier and Type	Interface and Description
static class	DataflowPipelineOptions.JobNameFactory Returns a normalized job name constructed from ApplicationNameOptions.getAppName(), the local system user name (if available), and the current time.

Nested classes/interfaces inherited from interface com.google.cloud.dataflow.sdk.options.DataflowPipelineDebugOptions

DataflowPipelineDebugOptions.DataflowClientFactory, DataflowPipelineDebugOptions.PathValidatorFactory, DataflowPipelineDebugOptions.StagerFactory

Nested classes/interfaces inherited from interface com.google.cloud.dataflow.sdk.options.DataflowPipelineWorkerPoolOptions

DataflowPipelineWorkerPoolOptions.AutoscalingAlgorithmType, DataflowPipelineWorkerPoolOptions.TeardownPolicy, DataflowPipelineWorkerPoolOptions.WorkerHarnessContainerImageFactory

Nested classes/interfaces inherited from interface com.google.cloud.dataflow.sdk.options.GcpOptions

GcpOptions.CredentialDirFactory, GcpOptions.DefaultProjectFactory, GcpOptions.GcpUserCredentialsFactory

Nested classes/interfaces inherited from interface com.google.cloud.dataflow.sdk.options.GoogleApiDebugOptions

GoogleApiDebugOptions.GoogleApiTracer

Nested classes/interfaces inherited from interface com.google.cloud.dataflow.sdk.options.PipelineOptions

PipelineOptions.CheckEnabled

Nested classes/interfaces inherited from interface com.google.cloud.dataflow.sdk.options.GcsOptions

GcsOptions.ExecutorServiceFactory

Nested classes/interfaces inherited from interface com.google.cloud.dataflow.sdk.options.DataflowWorkerLoggingOptions

DataflowWorkerLoggingOptions.Level, DataflowWorkerLoggingOptions.WorkerLogLevelOverrides

Nested classes/interfaces inherited from interface com.google.cloud.dataflow.sdk.options.DataflowProfilingOptions

DataflowProfilingOptions.DataflowProfilingAgentConfiguration

Method Summary

Modifier and Type	Method and Description
String	getJobName() The Dataflow job name is used as an idempotence key within the Dataflow service.
Map<String,String>	getLabels() Labels that will be applied to the billing records for this job.
String	getProject() Project id to use when launching jobs.
String	getStagingLocation() GCS path for staging local files, e.g.
boolean	getUpdate() Whether to update the currently running pipeline with the same name as this one.
void	setJobName(String value)
void	setLabels(Map<String,String> labels)
void	setProject(String value)
void	setStagingLocation(String value)
void	setUpdate(boolean value)

Methods inherited from interface com.google.cloud.dataflow.sdk.options.DataflowPipelineDebugOptions

getApiRootUrl, getDataflowClient, getDataflowEndpoint, getDataflowJobFile, getDumpHeapOnOOM, getExperiments, getNumberOfWorkerHarnessThreads, getOverrideWindmillBinary, getPathValidator, getPathValidatorClass, getPubsubRootUrl, getStager, getStagerClass, getTransformNameMapping, getWindmillServiceEndpoint, getWindmillServicePort, setApiRootUrl, setDataflowClient, setDataflowEndpoint, setDataflowJobFile, setDumpHeapOnOOM, setExperiments, setNumberOfWorkerHarnessThreads, setOverrideWindmillBinary, setPathValidator, setPathValidatorClass, setPubsubRootUrl, setStager, setStagerClass, setTransformNameMapping, setWindmillServiceEndpoint, setWindmillServicePort

Methods inherited from interface com.google.cloud.dataflow.sdk.options.DataflowPipelineWorkerPoolOptions

getAutoscalingAlgorithm, getDiskSizeGb, getFilesToStage, getMaxNumWorkers, getNetwork, getNumWorkers, getSubnetwork, getTeardownPolicy, getUsePublicIps, getWorkerDiskType, getWorkerHarnessContainerImage, getWorkerMachineType, getZone, setAutoscalingAlgorithm, setDiskSizeGb, setFilesToStage, setMaxNumWorkers, setNetwork, setNumWorkers, setSubnetwork, setTeardownPolicy, setUsePublicIps, setWorkerDiskType, setWorkerHarnessContainerImage, setWorkerMachineType, setZone

Methods inherited from interface com.google.cloud.dataflow.sdk.options.BigQueryOptions

getTempDatasetId, setTempDatasetId

Methods inherited from interface com.google.cloud.dataflow.sdk.options.StreamingOptions

isStreaming, setStreaming

Methods inherited from interface com.google.cloud.dataflow.sdk.options.ApplicationNameOptions

getAppName, setAppName

Methods inherited from interface com.google.cloud.dataflow.sdk.options.GcpOptions

getAuthorizationServerEncodedUrl, getCredentialDir, getCredentialFactoryClass, getCredentialId, getGcpCredential, getSecretsFile, getServiceAccountKeyfile, getServiceAccountName, getTokenServerUrl, setAuthorizationServerEncodedUrl, setCredentialDir, setCredentialFactoryClass, setCredentialId, setGcpCredential, setSecretsFile, setServiceAccountKeyfile, setServiceAccountName, setTokenServerUrl

Methods inherited from interface com.google.cloud.dataflow.sdk.options.GoogleApiDebugOptions

getGoogleApiTrace, setGoogleApiTrace

Methods inherited from interface com.google.cloud.dataflow.sdk.options.PipelineOptions

as, cloneAs, getRunner, getStableUniqueNames, getTempLocation, setRunner, setStableUniqueNames, setTempLocation

Methods inherited from interface com.google.cloud.dataflow.sdk.transforms.display.HasDisplayData

populateDisplayData

Methods inherited from interface com.google.cloud.dataflow.sdk.options.GcsOptions

getExecutorService, getGcsEndpoint, getGcsUploadBufferSizeBytes, getGcsUtil, setExecutorService, setGcsEndpoint, setGcsUploadBufferSizeBytes, setGcsUtil

Methods inherited from interface com.google.cloud.dataflow.sdk.options.CloudDebuggerOptions

getDebuggee, getEnableCloudDebugger, getMaxConditionCost, setDebuggee, setEnableCloudDebugger, setMaxConditionCost

Methods inherited from interface com.google.cloud.dataflow.sdk.options.DataflowWorkerLoggingOptions

getDefaultWorkerLogLevel, getWorkerLogLevelOverrides, setDefaultWorkerLogLevel, setWorkerLogLevelOverrides

Methods inherited from interface com.google.cloud.dataflow.sdk.options.DataflowProfilingOptions

getEnableProfilingAgent, getProfilingAgentConfiguration, setEnableProfilingAgent, setProfilingAgentConfiguration

Method Detail

getProject

@Validation.Required
 @Default.InstanceFactory(value=GcpOptions.DefaultProjectFactory.class)
String getProject()

Description copied from interface: GcpOptions

Project id to use when launching jobs.

Specified by:

getProject in interface GcpOptions

setProject

void setProject(String value)

Specified by:

setProject in interface GcpOptions

getStagingLocation

String getStagingLocation()

GCS path for staging local files, e.g. gs://bucket/object

Must be a valid Cloud Storage URL, beginning with the prefix "gs://"

At least one of PipelineOptions.getTempLocation() or getStagingLocation() must be set. If PipelineOptions.getTempLocation() is not set, then the Dataflow pipeline defaults to using getStagingLocation().

setStagingLocation

void setStagingLocation(String value)

getJobName

@Default.InstanceFactory(value=DataflowPipelineOptions.JobNameFactory.class)
String getJobName()

The Dataflow job name is used as an idempotence key within the Dataflow service. If there is an existing job that is currently active, another active job with the same name will not be able to be created. Defaults to using the ApplicationName-UserName-Date.

setJobName

void setJobName(String value)

getUpdate

boolean getUpdate()

Whether to update the currently running pipeline with the same name as this one.

Specified by:

getUpdate in interface DataflowPipelineDebugOptions

setUpdate

void setUpdate(boolean value)

Specified by:

setUpdate in interface DataflowPipelineDebugOptions

getLabels

Map<String,String> getLabels()

Labels that will be applied to the billing records for this job.

setLabels

void setLabels(Map<String,String> labels)