Class FileSystems
- java.lang.Object
-
- org.apache.beam.sdk.io.FileSystems
-
@Experimental(FILESYSTEM) public class FileSystems extends java.lang.Object
Clients facingFileSystem
utility.
-
-
Field Summary
Fields Modifier and Type Field Description static java.lang.String
DEFAULT_SCHEME
-
Constructor Summary
Constructors Constructor Description FileSystems()
-
Method Summary
All Methods Static Methods Concrete Methods Modifier and Type Method Description static void
copy(java.util.List<ResourceId> srcResourceIds, java.util.List<ResourceId> destResourceIds, MoveOptions... moveOptions)
Copies aList
of file-like resources from one location to another.static java.nio.channels.WritableByteChannel
create(ResourceId resourceId, java.lang.String mimeType)
Returns a write channel for the givenResourceId
.static java.nio.channels.WritableByteChannel
create(ResourceId resourceId, CreateOptions createOptions)
Returns a write channel for the givenResourceId
withCreateOptions
.static void
delete(java.util.Collection<ResourceId> resourceIds, MoveOptions... moveOptions)
Deletes a collection of resources.static boolean
hasGlobWildcard(java.lang.String spec)
Checks whether the given spec contains a glob wildcard character.static MatchResult
match(java.lang.String spec)
Likematch(List)
, but for a single resource specification.static MatchResult
match(java.lang.String spec, EmptyMatchTreatment emptyMatchTreatment)
Likematch(String)
, but with a configurableEmptyMatchTreatment
.static java.util.List<MatchResult>
match(java.util.List<java.lang.String> specs)
This is the entry point to convert user-provided specs toResourceIds
.static java.util.List<MatchResult>
match(java.util.List<java.lang.String> specs, EmptyMatchTreatment emptyMatchTreatment)
Likematch(List)
, but with a configurableEmptyMatchTreatment
.static ResourceId
matchNewResource(java.lang.String singleResourceSpec, boolean isDirectory)
Returns a newResourceId
that represents the named resource of a type corresponding to the resource type.static java.util.List<MatchResult>
matchResources(java.util.List<ResourceId> resourceIds)
ReturnsMatchResults
for the givenresourceIds
.static MatchResult.Metadata
matchSingleFileSpec(java.lang.String spec)
Returns theMatchResult.Metadata
for a single file resource.static java.nio.channels.ReadableByteChannel
open(ResourceId resourceId)
Returns a read channel for the givenResourceId
.static void
rename(java.util.List<ResourceId> srcResourceIds, java.util.List<ResourceId> destResourceIds, MoveOptions... moveOptions)
Renames aList
of file-like resources from one location to another.static void
setDefaultPipelineOptions(PipelineOptions options)
Sets the default configuration in workers.
-
-
-
Field Detail
-
DEFAULT_SCHEME
public static final java.lang.String DEFAULT_SCHEME
- See Also:
- Constant Field Values
-
-
Method Detail
-
hasGlobWildcard
public static boolean hasGlobWildcard(java.lang.String spec)
Checks whether the given spec contains a glob wildcard character.
-
match
public static java.util.List<MatchResult> match(java.util.List<java.lang.String> specs) throws java.io.IOException
This is the entry point to convert user-provided specs toResourceIds
. Callers should usematch(java.util.List<java.lang.String>)
to resolve users specs ambiguities before calling other methods.Implementation handles the following ambiguities of a user-provided spec:
spec
could be a glob or a uri.match(java.util.List<java.lang.String>)
should be able to tell and choose efficient implementations.- The user-provided
spec
might refer to files or directories. It is common that users that wish to indicate a directory will omit the trailing path delimiter, such as"/tmp/dir"
in Linux. TheFileSystem
should be able to recognize a directory with the trailing path delimiter omitted, but should always return a correctResourceId
(e.g.,"/tmp/dir/"
inside the returnedMatchResult
.
All
FileSystem
implementations should support glob in the final hierarchical path component ofResourceId
. This allows SDK libraries to construct file system agnostic spec.FileSystems
can support additional patterns for user-provided specs.In case the spec schemes don't match any known
FileSystem
implementations, FileSystems will attempt to useLocalFileSystem
to resolve a path.Specs that do not match any resources are treated according to
EmptyMatchTreatment.DISALLOW
.- Returns:
List<MatchResult>
in the same order of the input specs.- Throws:
java.lang.IllegalArgumentException
- if specs are invalid -- empty or have different schemes.java.io.IOException
- if all specs failed to match due to issues like: network connection, authorization. Exception for individual spec is deferred until callers retrieve metadata withMatchResult.metadata()
.
-
match
public static java.util.List<MatchResult> match(java.util.List<java.lang.String> specs, EmptyMatchTreatment emptyMatchTreatment) throws java.io.IOException
Likematch(List)
, but with a configurableEmptyMatchTreatment
.- Throws:
java.io.IOException
-
match
public static MatchResult match(java.lang.String spec) throws java.io.IOException
Likematch(List)
, but for a single resource specification.The function
match(List)
is preferred when matching multiple patterns, as it allows for bulk API calls to remote filesystems.- Throws:
java.io.IOException
-
match
public static MatchResult match(java.lang.String spec, EmptyMatchTreatment emptyMatchTreatment) throws java.io.IOException
Likematch(String)
, but with a configurableEmptyMatchTreatment
.- Throws:
java.io.IOException
-
matchSingleFileSpec
public static MatchResult.Metadata matchSingleFileSpec(java.lang.String spec) throws java.io.IOException
Returns theMatchResult.Metadata
for a single file resource. Expects a resource specificationspec
that matches a single result.- Parameters:
spec
- a resource specification that matches exactly one result.- Returns:
- the
MatchResult.Metadata
for the specified resource. - Throws:
java.io.FileNotFoundException
- if the file resource is not found.java.io.IOException
- in the event of an error in the inner call tomatch(java.util.List<java.lang.String>)
, or if the given spec does not match exactly 1 result.
-
matchResources
public static java.util.List<MatchResult> matchResources(java.util.List<ResourceId> resourceIds) throws java.io.IOException
ReturnsMatchResults
for the givenresourceIds
.- Parameters:
resourceIds
-resourceIds
that might be derived frommatch(java.util.List<java.lang.String>)
,ResourceId.resolve(java.lang.String, org.apache.beam.sdk.io.fs.ResolveOptions)
, orResourceId.getCurrentDirectory()
.- Throws:
java.io.IOException
- if allresourceIds
failed to match due to issues like: network connection, authorization. Exception for individualResourceId
need to be deferred until callers retrieve metadata withMatchResult.metadata()
.
-
create
public static java.nio.channels.WritableByteChannel create(ResourceId resourceId, java.lang.String mimeType) throws java.io.IOException
Returns a write channel for the givenResourceId
.The resource is not expanded; it is used verbatim.
- Parameters:
resourceId
- the reference of the file-like resource to createmimeType
- the mine type of the file-like resource to create- Throws:
java.io.IOException
-
create
public static java.nio.channels.WritableByteChannel create(ResourceId resourceId, CreateOptions createOptions) throws java.io.IOException
Returns a write channel for the givenResourceId
withCreateOptions
.The resource is not expanded; it is used verbatim.
- Parameters:
resourceId
- the reference of the file-like resource to createcreateOptions
- the configuration of the create operation- Throws:
java.io.IOException
-
open
public static java.nio.channels.ReadableByteChannel open(ResourceId resourceId) throws java.io.IOException
Returns a read channel for the givenResourceId
.The resource is not expanded; it is used verbatim.
If seeking is supported, then this returns a
SeekableByteChannel
.- Parameters:
resourceId
- the reference of the file-like resource to open- Throws:
java.io.IOException
-
copy
public static void copy(java.util.List<ResourceId> srcResourceIds, java.util.List<ResourceId> destResourceIds, MoveOptions... moveOptions) throws java.io.IOException
Copies aList
of file-like resources from one location to another.The number of source resources must equal the number of destination resources. Destination resources will be created recursively.
srcResourceIds
anddestResourceIds
must have the same scheme.It doesn't support copying globs.
- Parameters:
srcResourceIds
- the references of the source resourcesdestResourceIds
- the references of the destination resources- Throws:
java.io.IOException
-
rename
public static void rename(java.util.List<ResourceId> srcResourceIds, java.util.List<ResourceId> destResourceIds, MoveOptions... moveOptions) throws java.io.IOException
Renames aList
of file-like resources from one location to another.The number of source resources must equal the number of destination resources. Destination resources will be created recursively.
srcResourceIds
anddestResourceIds
must have the same scheme.It doesn't support renaming globs.
Src files will be removed, even if the copy is skipped due to specified move options.
- Parameters:
srcResourceIds
- the references of the source resourcesdestResourceIds
- the references of the destination resources- Throws:
java.io.IOException
-
delete
public static void delete(java.util.Collection<ResourceId> resourceIds, MoveOptions... moveOptions) throws java.io.IOException
Deletes a collection of resources.resourceIds
must have the same scheme.- Parameters:
resourceIds
- the references of the resources to delete.- Throws:
java.io.IOException
-
setDefaultPipelineOptions
@Internal public static void setDefaultPipelineOptions(PipelineOptions options)
Sets the default configuration in workers.It will be used in
FileSystemRegistrars
for all schemes.This is expected only to be used by runners after
Pipeline.run
, or in tests.
-
matchNewResource
public static ResourceId matchNewResource(java.lang.String singleResourceSpec, boolean isDirectory)
Returns a newResourceId
that represents the named resource of a type corresponding to the resource type.The supplied
singleResourceSpec
is expected to be in a proper format, including any necessary escaping, for the underlyingFileSystem
.This function may throw an
IllegalArgumentException
if given an invalid argument, such as when the specifiedsingleResourceSpec
is not a valid resource name.
-
-