Interface PipelineOptions
-
- All Superinterfaces:
HasDisplayData
- All Known Subinterfaces:
ApplicationNameOptions
,ExecutorOptions
,ExperimentalOptions
,FileStagingOptions
,ManualDockerEnvironmentOptions
,MetricsOptions
,PortablePipelineOptions
,RemoteEnvironmentOptions
,ResourceHintsOptions
,SdkHarnessOptions
,StreamingOptions
,TestPipeline.TestValueProviderOptions
,TestPipelineOptions
@ThreadSafe public interface PipelineOptions extends HasDisplayData
PipelineOptions are used to configure Pipelines. You can extendPipelineOptions
to create custom configuration options specific to yourPipeline
, for both local execution and execution via aPipelineRunner
.PipelineOptions
and their subinterfaces represent a collection of properties which can be manipulated in a type safe manner.PipelineOptions
is backed by a dynamicProxy
which allows for type safe manipulation of properties in an extensible fashion through plain old Java interfaces.PipelineOptions
can be created withPipelineOptionsFactory.create()
andPipelineOptionsFactory.as(Class)
. They can be created from command-line arguments withPipelineOptionsFactory.fromArgs(String[])
. They can be converted to another type by invokingas(Class)
and can be accessed from within aDoFn
by invokinggetPipelineOptions()
on the inputContext
object.Please don't implement
PipelineOptions
, it implies that it is backwards-incompatible to add new options. User-implementedPipelineOptions
is not accepted byPipeline
.For example:
// The most common way to construct PipelineOptions is via command-line argument parsing: public static void main(String[] args) { // Will parse the arguments passed into the application and construct a PipelineOptions // Note that --help will print registered options, and --help=PipelineOptionsClassName // will print out usage for the specific class. PipelineOptions options = PipelineOptionsFactory.fromArgs(args).create(); Pipeline p = Pipeline.create(options); ... p.run(); } // To create options for the DirectRunner: DirectOptions directRunnerOptions = PipelineOptionsFactory.as(DirectOptions.class); // To cast from one type to another using the as(Class) method: ApplicationNameOptions applicationNameOptions = directPipelineOptions.as(ApplicationNameOptions.class); // Options for the same property are shared between types // The statement below will print out the name of the enclosing class by default System.out.println(applicationNameOptions.getApplicationName()); // Prints out registered options. PipelineOptionsFactory.printHelp(System.out); // Prints out options which are available to be set on ApplicationNameOptions PipelineOptionsFactory.printHelp(System.out, ApplicationNameOptions.class);
Defining Your Own PipelineOptions
Defining your own
PipelineOptions
is the way for you to make configuration options available for both local execution and execution via aPipelineRunner
. By having PipelineOptionsFactory as your command-line interpreter, you will provide a standardized way for users to interact with your application via the command-line.To define your own
PipelineOptions
, you create a public interface which extendsPipelineOptions
and define getter/setter pairs. These getter/setter pairs define a collection of JavaBean properties.For example:
// Creates a user defined property called "myProperty" public interface MyOptions extends PipelineOptions { String getMyProperty(); void setMyProperty(String value); }
Note: Please see the section on Registration below when using custom property types.
Restrictions
Since PipelineOptions can be "cast" to multiple types dynamically using
as(Class)
, a property must conform to the following set of restrictions:- Any property with the same name must have the same return type for all derived interfaces
of
PipelineOptions
. - Every bean property of any interface derived from
PipelineOptions
must have a getter and setter method. - Every method must conform to being a getter or setter for a JavaBean.
- The derived interface of
PipelineOptions
must be composable with every interface part registered with the PipelineOptionsFactory. - Only getters may be annotated with
@JsonIgnore
. - If any getter is annotated with
@JsonIgnore
, then all getters for this property must be annotated with@JsonIgnore
. - If any getter is annotated with
JsonDeserialize
andJsonSerialize
, then all getters for this property must also be.
Annotations For PipelineOptions
@Description
can be used to annotate an interface or a getter with useful information which is output when--help
is invoked viaPipelineOptionsFactory.fromArgs(String[])
.@Default
represents a set of annotations that can be used to annotate getter properties onPipelineOptions
with information representing the default value to be returned if no value is specified. Any default implementation (using thedefault
keyword) is ignored.@Hidden
hides an option from being listed when--help
is invoked viaPipelineOptionsFactory.fromArgs(String[])
.@Validation
represents a set of annotations that can be used to annotate getter properties onPipelineOptions
with information representing the validation criteria to be used when validating with thePipelineOptionsValidator
. Validation will be performed if during construction of thePipelineOptions
,PipelineOptionsFactory.withValidation()
is invoked.@JsonIgnore
is used to prevent a property from being serialized and available during execution ofDoFn
. See the Serialization section below for more details.@JsonSerialize
and@JsonDeserialize
is used to control how a property is (de)serialized when the PipelineOptions are (de)serialized to JSON. See the Serialization section below for more details.Registration Of PipelineOptions
Registration of
PipelineOptions
by an application guarantees that thePipelineOptions
is composable during execution of theirPipeline
and meets the restrictions listed above or will fail during registration. Registration also lists the registeredPipelineOptions
when--help
is invoked viaPipelineOptionsFactory.fromArgs(String[])
.Registration can be performed by invoking
PipelineOptionsFactory.register(java.lang.Class<? extends org.apache.beam.sdk.options.PipelineOptions>)
within a users application or via automatic registration by creating aServiceLoader
entry and a concrete implementation of thePipelineOptionsRegistrar
interface.It is optional but recommended to use one of the many build time tools such as
AutoService
to generate the necessary META-INF files automatically.A list of registered options can be fetched from
PipelineOptionsFactory.getRegisteredOptions()
.Serialization Of PipelineOptions
PipelineOptions
is intentionally not markedSerializable
, in order to discourage pipeline authors from capturingPipelineOptions
at pipeline construction time, because a pipeline may be saved as a template and run with a different set of options than the ones it was constructed with. SeePipeline.run(PipelineOptions)
.However,
PipelineRunner
s require support for options to be serialized. Each property withinPipelineOptions
must be able to be serialized using Jackson'sObjectMapper
or the getter method for the property annotated with@JsonIgnore
.Jackson supports serialization of many types and supports a useful set of annotations to aid in serialization of custom types. We point you to the public Jackson documentation when attempting to add serialization support for your custom types. Note that
PipelineOptions
relies on Jackson's ability to automatically configure theObjectMapper
with additional modules viaObjectMapper.findModules()
.To further customize serialization, getter methods may be annotated with
@JsonSerialize
and@JsonDeserialize
.@JsonDeserialize
is also used when parsing command line arguments.Note: A property must be annotated with BOTH
@JsonDeserialize
and@JsonSerialize
or neither. It is an error to have a property annotated with only@JsonDeserialize
or@JsonSerialize
.Note: It is an error to have the same property available in multiple interfaces with only some of them being annotated with
@JsonIgnore
. It is also an error to mark a setter for a property with@JsonIgnore
.
-
-
Nested Class Summary
Nested Classes Modifier and Type Interface Description static class
PipelineOptions.AtomicLongFactory
DefaultValueFactory
which supplies an ID that is guaranteed to be unique within the given process.static class
PipelineOptions.CheckEnabled
Enumeration of the possible states for a given check.static class
PipelineOptions.DirectRunner
ADefaultValueFactory
that obtains the class of theDirectRunner
if it exists on the classpath, and throws an exception otherwise.static class
PipelineOptions.JobNameFactory
Returns a normalized job name constructed fromApplicationNameOptions.getAppName()
, the local system user name (if available), the current time, and a random integer.static class
PipelineOptions.UserAgentFactory
Returns a user agent string constructed fromReleaseInfo.getName()
andReleaseInfo.getVersion()
, in the format[name]/[version]
.
-
Method Summary
All Methods Instance Methods Abstract Methods Modifier and Type Method Description <T extends PipelineOptions>
Tas(java.lang.Class<T> kls)
Transforms this object into an object of type<T>
saving each property that has been manipulated.java.lang.String
getJobName()
long
getOptionsId()
Provides a process wide unique ID for thisPipelineOptions
object, assigned at graph construction time.java.lang.Class<? extends PipelineRunner<?>>
getRunner()
The pipeline runner that will be used to execute the pipeline.PipelineOptions.CheckEnabled
getStableUniqueNames()
Whether to check for stable unique names on each transform.java.lang.String
getTempLocation()
A pipeline level default location for storing temporary files.java.lang.String
getUserAgent()
A user agent string as per RFC2616, describing the pipeline to external services.java.util.Map<java.lang.String,java.util.Map<java.lang.String,java.lang.Object>>
outputRuntimeOptions()
Returns a map of properties which correspond toValueProvider.RuntimeValueProvider
, keyed by the property name.void
setJobName(java.lang.String jobName)
void
setOptionsId(long id)
void
setRunner(java.lang.Class<? extends PipelineRunner<?>> kls)
void
setStableUniqueNames(PipelineOptions.CheckEnabled enabled)
void
setTempLocation(java.lang.String value)
void
setUserAgent(java.lang.String userAgent)
-
Methods inherited from interface org.apache.beam.sdk.transforms.display.HasDisplayData
populateDisplayData
-
-
-
-
Method Detail
-
as
<T extends PipelineOptions> T as(java.lang.Class<T> kls)
Transforms this object into an object of type<T>
saving each property that has been manipulated.<T>
must extendPipelineOptions
.If
<T>
is not registered with thePipelineOptionsFactory
, then we attempt to verify that<T>
is composable with every interface that this instance of thePipelineOptions
has seen.- Parameters:
kls
- The class of the type to transform to.- Returns:
- An object of type kls.
-
getRunner
@Required @InstanceFactory(DirectRunner.class) java.lang.Class<? extends PipelineRunner<?>> getRunner()
The pipeline runner that will be used to execute the pipeline. For registered runners, the class name can be specified, otherwise the fully qualified name needs to be specified.
-
setRunner
void setRunner(java.lang.Class<? extends PipelineRunner<?>> kls)
-
getStableUniqueNames
@Required @Enum("WARNING") PipelineOptions.CheckEnabled getStableUniqueNames()
Whether to check for stable unique names on each transform. This is necessary to support updating of pipelines.
-
setStableUniqueNames
void setStableUniqueNames(PipelineOptions.CheckEnabled enabled)
-
getTempLocation
java.lang.String getTempLocation()
A pipeline level default location for storing temporary files.This can be a path of any file system.
getTempLocation()
can be used as a default location in otherPipelineOptions
.If it is unset,
PipelineRunner
can override it.
-
setTempLocation
void setTempLocation(java.lang.String value)
-
getJobName
@InstanceFactory(JobNameFactory.class) java.lang.String getJobName()
-
setJobName
void setJobName(java.lang.String jobName)
-
outputRuntimeOptions
java.util.Map<java.lang.String,java.util.Map<java.lang.String,java.lang.Object>> outputRuntimeOptions()
Returns a map of properties which correspond toValueProvider.RuntimeValueProvider
, keyed by the property name. The value is a map containing type and default information.
-
getOptionsId
@Hidden @InstanceFactory(AtomicLongFactory.class) long getOptionsId()
Provides a process wide unique ID for thisPipelineOptions
object, assigned at graph construction time.
-
setOptionsId
void setOptionsId(long id)
-
getUserAgent
@InstanceFactory(UserAgentFactory.class) java.lang.String getUserAgent()
A user agent string as per RFC2616, describing the pipeline to external services.https://www.ietf.org/rfc/rfc2616.txt
It should follow the BNF Form:
Where a token is a series of characters without a separator.user agent = 1*(product | comment) product = token ["/" product-version] product-version = token
The string defaults to
[name]/[version]
based on the properties of the Apache Beam release.
-
setUserAgent
void setUserAgent(java.lang.String userAgent)
-
-