Package alluxio.underfs
Class BaseUnderFileSystem
- java.lang.Object
-
- alluxio.underfs.BaseUnderFileSystem
-
- All Implemented Interfaces:
AsyncUfsClient
,UnderFileSystem
,java.io.Closeable
,java.lang.AutoCloseable
- Direct Known Subclasses:
ConsistentUnderFileSystem
,ObjectUnderFileSystem
@ThreadSafe public abstract class BaseUnderFileSystem extends java.lang.Object implements UnderFileSystem
A base abstractUnderFileSystem
.
-
-
Nested Class Summary
-
Nested classes/interfaces inherited from interface alluxio.underfs.UnderFileSystem
UnderFileSystem.Factory, UnderFileSystem.SpaceType
-
-
Field Summary
Fields Modifier and Type Field Description static Pair<AccessControlList,DefaultAccessControlList>
EMPTY_ACL
protected UnderFileSystemConfiguration
mUfsConf
UFS Configuration options.protected AlluxioURI
mUri
The UFSAlluxioURI
used to create thisBaseUnderFileSystem
.
-
Constructor Summary
Constructors Modifier Constructor Description protected
BaseUnderFileSystem(AlluxioURI uri, UnderFileSystemConfiguration ufsConf)
Constructs anBaseUnderFileSystem
.
-
Method Summary
All Methods Static Methods Instance Methods Concrete Methods Modifier and Type Method Description void
close()
java.io.OutputStream
create(java.lang.String path)
Creates a file in the under file system with the indicated name.boolean
deleteDirectory(java.lang.String path)
Deletes a directory from the under file system with the indicated name non-recursively.boolean
exists(java.lang.String path)
Checks if a file or directory exists in under file system.Pair<AccessControlList,DefaultAccessControlList>
getAclPair(java.lang.String path)
Gets the ACL and the Default ACL of a file or directory in under file system.SyncInfo
getActiveSyncInfo()
Return the active sync info for the specified syncPoints.AlluxioConfiguration
getConfiguration()
Gets the under file system configuration.java.lang.String
getFingerprint(java.lang.String path)
Computes and returns a fingerprint for the path.UfsMode
getOperationMode(java.util.Map<java.lang.String,UfsMode> physicalUfsState)
AnUnderFileSystem
may be composed of one or more "physical UFS"s.Fingerprint
getParsedFingerprint(java.lang.String path)
Computes and returns a fingerprint for the path.Fingerprint
getParsedFingerprint(java.lang.String path, java.lang.String contentHash)
Same asUnderFileSystem.getParsedFingerprint(String)
except, will use the given content hash as theFingerprint.Tag.CONTENT_HASH
field of the fingerprint if non-null.java.util.List<java.lang.String>
getPhysicalStores()
AnUnderFileSystem
may be composed of one or more "physical UFS"s.boolean
isObjectStorage()
boolean
isSeekable()
Denotes if the under storage supports seeking.UfsStatus[]
listStatus(java.lang.String path, ListOptions options)
Returns an array of statuses of the files and directories in the directory denoted by this abstract pathname, with options.java.util.Iterator<UfsStatus>
listStatusIterable(java.lang.String path, ListOptions options, java.lang.String startAfter, int batchSize)
Lists the ufs statuses iteratively.boolean
mkdirs(java.lang.String path)
Creates the directory named by this abstract pathname.java.io.InputStream
open(java.lang.String path)
Opens anInputStream
for a file in under filesystem at the indicated path.void
performListingAsync(java.lang.String path, java.lang.String continuationToken, java.lang.String startAfter, DescendantType descendantType, boolean checkStatus, java.util.function.Consumer<UfsLoadResult> onComplete, java.util.function.Consumer<java.lang.Throwable> onError)
Lists the ufs statuses for a given path.AlluxioURI
resolveUri(AlluxioURI ufsBaseUri, java.lang.String alluxioPath)
Returns anAlluxioURI
representation for theUnderFileSystem
given a base UFS URI, and the Alluxio path from the base.void
setAclEntries(java.lang.String path, java.util.List<AclEntry> aclEntries)
Sets the access control list of a file or directory in under file system.boolean
startActiveSyncPolling(long txId)
Start Active Sync.void
startSync(AlluxioURI uri)
Add Sync Point.boolean
stopActiveSyncPolling()
Stop Active Sync.void
stopSync(AlluxioURI uri)
Stop Sync Point.boolean
supportsActiveSync()
Whether this type of UFS supports active sync.protected static java.lang.String
validatePath(java.lang.String path)
Clean the path by creating a URI and turning it back to a string.-
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
-
Methods inherited from interface alluxio.underfs.UnderFileSystem
cleanup, connectFromMaster, connectFromWorker, create, createNonexistingFile, createNonexistingFile, deleteDirectory, deleteExistingDirectory, deleteExistingDirectory, deleteExistingFile, deleteFile, getBlockSizeByte, getDirectoryStatus, getExistingDirectoryStatus, getExistingFileStatus, getExistingStatus, getFileLocations, getFileLocations, getFileStatus, getFileStatus, getSpace, getStatus, getStatus, getTemporaryToken, getUnderFSType, isDirectory, isExistingDirectory, isFile, isTemporaryConnectionToken, listStatus, listStatuses, mkdirs, open, openExistingFile, openExistingFile, openPositionRead, refreshTemporaryToken, renameDirectory, renameFile, renameRenamableDirectory, renameRenamableFile, setMode, setOwner, setTemporaryTokenRefreshContext, supportsFlush
-
-
-
-
Field Detail
-
EMPTY_ACL
public static final Pair<AccessControlList,DefaultAccessControlList> EMPTY_ACL
-
mUri
protected final AlluxioURI mUri
The UFSAlluxioURI
used to create thisBaseUnderFileSystem
.
-
mUfsConf
protected final UnderFileSystemConfiguration mUfsConf
UFS Configuration options.
-
-
Constructor Detail
-
BaseUnderFileSystem
protected BaseUnderFileSystem(AlluxioURI uri, UnderFileSystemConfiguration ufsConf)
Constructs anBaseUnderFileSystem
.- Parameters:
uri
- theAlluxioURI
used to create this ufsufsConf
- UFS configuration
-
-
Method Detail
-
close
public void close() throws java.io.IOException
- Specified by:
close
in interfacejava.lang.AutoCloseable
- Specified by:
close
in interfacejava.io.Closeable
- Throws:
java.io.IOException
-
create
public java.io.OutputStream create(java.lang.String path) throws java.io.IOException
Description copied from interface:UnderFileSystem
Creates a file in the under file system with the indicated name. Parents directories will be created recursively.- Specified by:
create
in interfaceUnderFileSystem
- Parameters:
path
- the file name- Returns:
- A
OutputStream
object - Throws:
java.io.IOException
-
deleteDirectory
public boolean deleteDirectory(java.lang.String path) throws java.io.IOException
Description copied from interface:UnderFileSystem
Deletes a directory from the under file system with the indicated name non-recursively. A non-recursive delete is successful only if the directory is empty.- Specified by:
deleteDirectory
in interfaceUnderFileSystem
- Parameters:
path
- of the directory to delete- Returns:
- true if directory was found and deleted, false otherwise
- Throws:
java.io.IOException
-
exists
public boolean exists(java.lang.String path) throws java.io.IOException
Description copied from interface:UnderFileSystem
Checks if a file or directory exists in under file system.- Specified by:
exists
in interfaceUnderFileSystem
- Parameters:
path
- the absolute path- Returns:
- true if the path exists, false otherwise
- Throws:
java.io.IOException
-
getAclPair
public Pair<AccessControlList,DefaultAccessControlList> getAclPair(java.lang.String path) throws java.io.IOException
Description copied from interface:UnderFileSystem
Gets the ACL and the Default ACL of a file or directory in under file system.- Specified by:
getAclPair
in interfaceUnderFileSystem
- Parameters:
path
- the path to the file or directory- Returns:
- the access control list, along with a Default ACL if it is a directory return null if ACL is unsupported or disabled
- Throws:
java.io.IOException
- if ACL is supported and enabled but cannot be retrieved
-
setAclEntries
public void setAclEntries(java.lang.String path, java.util.List<AclEntry> aclEntries) throws java.io.IOException
Description copied from interface:UnderFileSystem
Sets the access control list of a file or directory in under file system. if the ufs does not support acls, this is a noop. This will overwrite the ACL and defaultACL in the UFS.- Specified by:
setAclEntries
in interfaceUnderFileSystem
- Parameters:
path
- the path to the file or directoryaclEntries
- the access control list + default acl represented in a list of acl entries- Throws:
java.io.IOException
-
getConfiguration
public AlluxioConfiguration getConfiguration()
Description copied from interface:UnderFileSystem
Gets the under file system configuration.- Specified by:
getConfiguration
in interfaceUnderFileSystem
- Returns:
- the configuration
-
getFingerprint
public java.lang.String getFingerprint(java.lang.String path)
Description copied from interface:UnderFileSystem
Computes and returns a fingerprint for the path. The fingerprint is used to determine if two UFS files are identical. The fingerprint must be deterministic, and must not change if a file is only renamed (identical content and permissions). ReturnsConstants.INVALID_UFS_FINGERPRINT
if there is any error.- Specified by:
getFingerprint
in interfaceUnderFileSystem
- Parameters:
path
- the path to compute the fingerprint for- Returns:
- the string representing the fingerprint
-
getParsedFingerprint
public Fingerprint getParsedFingerprint(java.lang.String path)
Description copied from interface:UnderFileSystem
Computes and returns a fingerprint for the path. The fingerprint is used to determine if two UFS files are identical. The fingerprint must be deterministic, and must not change if a file is only renamed (identical content and permissions). ReturnsFingerprint.INVALID_FINGERPRINT
if there is any error. The default implementation relies onUnderFileSystem.getFingerprint(String)
and there is one extra parsing. This default implementation is mainly for backward compatibility. Override this for performance.- Specified by:
getParsedFingerprint
in interfaceUnderFileSystem
- Parameters:
path
- the path to compute the fingerprint for- Returns:
- the string representing the fingerprint
-
getParsedFingerprint
public Fingerprint getParsedFingerprint(java.lang.String path, @Nullable java.lang.String contentHash)
Description copied from interface:UnderFileSystem
Same asUnderFileSystem.getParsedFingerprint(String)
except, will use the given content hash as theFingerprint.Tag.CONTENT_HASH
field of the fingerprint if non-null. This is intended to be used when the file is already in Alluxio and a fingerprint is being created based on that file where the content hash has already been computed.- Specified by:
getParsedFingerprint
in interfaceUnderFileSystem
- Parameters:
path
- the path to compute the fingerprint forcontentHash
- is used as theFingerprint.Tag.CONTENT_HASH
field when creating the fingerprint.- Returns:
- the string representing the fingerprint
-
getOperationMode
public UfsMode getOperationMode(java.util.Map<java.lang.String,UfsMode> physicalUfsState)
Description copied from interface:UnderFileSystem
AnUnderFileSystem
may be composed of one or more "physical UFS"s. This method is used to determine the operation mode based on the physical UFS operation modes. For example, if thisUnderFileSystem
is composed of physical UFS hdfs://ns1/ and hdfs://ns2/ with read operations split b/w the two, with physicalUfsState{hdfs://ns1/:NO_ACCESS, hdfs://ns2/:READ_WRITE} this method can return READ_ONLY to allow reads to proceed from hdfs://ns2/.- Specified by:
getOperationMode
in interfaceUnderFileSystem
- Parameters:
physicalUfsState
- the state of physical UFSs for thisUnderFileSystem
; keys are expected to be normalized (ending with /)- Returns:
- the desired operation mode for this UFS
-
getPhysicalStores
public java.util.List<java.lang.String> getPhysicalStores()
Description copied from interface:UnderFileSystem
AnUnderFileSystem
may be composed of one or more "physical UFS"s. This method returns all underlying physical stores; normalized with only scheme and authority.- Specified by:
getPhysicalStores
in interfaceUnderFileSystem
- Returns:
- physical UFSs this
UnderFileSystem
is composed of
-
isObjectStorage
public boolean isObjectStorage()
- Specified by:
isObjectStorage
in interfaceUnderFileSystem
- Returns:
- true if under storage is an object store, false otherwise
-
isSeekable
public boolean isSeekable()
Description copied from interface:UnderFileSystem
Denotes if the under storage supports seeking. Note, the under file system subclass that returns true for this method should return the input stream extendingSeekableUnderFileInputStream
in theUnderFileSystem.open(String, OpenOptions)
method.- Specified by:
isSeekable
in interfaceUnderFileSystem
- Returns:
- true if under storage is seekable, false otherwise
-
listStatusIterable
@Nullable public java.util.Iterator<UfsStatus> listStatusIterable(java.lang.String path, ListOptions options, java.lang.String startAfter, int batchSize) throws java.io.IOException
Description copied from interface:UnderFileSystem
Lists the ufs statuses iteratively.- Specified by:
listStatusIterable
in interfaceUnderFileSystem
- Parameters:
path
- the abstract pathname to listoptions
- for list directorystartAfter
- the start after tokenbatchSize
- the batch size- Returns:
- An iterator of ufs status. Returns
null
if this abstract pathname does not denote a directory. - Throws:
java.io.IOException
-
performListingAsync
public void performListingAsync(java.lang.String path, @Nullable java.lang.String continuationToken, @Nullable java.lang.String startAfter, DescendantType descendantType, boolean checkStatus, java.util.function.Consumer<UfsLoadResult> onComplete, java.util.function.Consumer<java.lang.Throwable> onError)
Description copied from interface:AsyncUfsClient
Lists the ufs statuses for a given path. TheUfsStatus.getName()
function for the returned values should include the full path of each item from the UFS root (not including the bucket name for object stores). It differs from a traditional listing in that if the input variable checkStatus is true, theUfsStatus
for the base path should be included at the start of the results. The function should return immediately, and perform the operation asynchronously.- Specified by:
performListingAsync
in interfaceAsyncUfsClient
- Parameters:
path
- the path in ufscontinuationToken
- the continuation tokenstartAfter
- the start after string where the loading starts fromdescendantType
- the load descendant type (NONE/ONE/ALL)checkStatus
- if true the call will perform a GetStatus on the path to see if an object exists, which should be returned as part of the resultonComplete
- the callback when the load is completeonError
- the callback when the load encountered an error
-
listStatus
@Nullable public UfsStatus[] listStatus(java.lang.String path, ListOptions options) throws java.io.IOException
Description copied from interface:UnderFileSystem
Returns an array of statuses of the files and directories in the directory denoted by this abstract pathname, with options.If this abstract pathname does not denote a directory, then this method returns
null
. Otherwise an array of statuses is returned, one for each file or directory. Names denoting the directory itself and the directory's parent directory are not included in the result. Each string is a path relative to the given directory.There is no guarantee that the name strings in the resulting array will appear in any specific order; they are not, in particular, guaranteed to appear in alphabetical order.
- Specified by:
listStatus
in interfaceUnderFileSystem
- Parameters:
path
- the abstract pathname to listoptions
- for list directory- Returns:
- An array of statuses naming the files and directories in the directory denoted by this
abstract pathname. The array will be empty if the directory is empty. Returns
null
if this abstract pathname does not denote a directory. - Throws:
java.io.IOException
-
open
public java.io.InputStream open(java.lang.String path) throws java.io.IOException
Description copied from interface:UnderFileSystem
Opens anInputStream
for a file in under filesystem at the indicated path.- Specified by:
open
in interfaceUnderFileSystem
- Parameters:
path
- the file name- Returns:
- The
InputStream
object - Throws:
java.io.IOException
-
mkdirs
public boolean mkdirs(java.lang.String path) throws java.io.IOException
Description copied from interface:UnderFileSystem
Creates the directory named by this abstract pathname. If the folder already exists, the method returns false. The method creates any necessary but nonexistent parent directories.- Specified by:
mkdirs
in interfaceUnderFileSystem
- Parameters:
path
- the folder to create- Returns:
true
if and only if the directory was created;false
otherwise- Throws:
java.io.IOException
-
resolveUri
public AlluxioURI resolveUri(AlluxioURI ufsBaseUri, java.lang.String alluxioPath)
Description copied from interface:UnderFileSystem
Returns anAlluxioURI
representation for theUnderFileSystem
given a base UFS URI, and the Alluxio path from the base. The default implementation simply concatenates the path to the base URI. This should be overridden if a subclass needs alternate functionality.- Specified by:
resolveUri
in interfaceUnderFileSystem
- Parameters:
ufsBaseUri
- the baseAlluxioURI
in the ufsalluxioPath
- the path in Alluxio from the given base- Returns:
- the UFS
AlluxioURI
representing the Alluxio path
-
supportsActiveSync
public boolean supportsActiveSync()
Description copied from interface:UnderFileSystem
Whether this type of UFS supports active sync.- Specified by:
supportsActiveSync
in interfaceUnderFileSystem
- Returns:
- true if this type of UFS supports active sync, false otherwise
-
getActiveSyncInfo
public SyncInfo getActiveSyncInfo()
Description copied from interface:UnderFileSystem
Return the active sync info for the specified syncPoints.- Specified by:
getActiveSyncInfo
in interfaceUnderFileSystem
- Returns:
- active sync info consisting of what changed for these sync points
-
startActiveSyncPolling
public boolean startActiveSyncPolling(long txId) throws java.io.IOException
Description copied from interface:UnderFileSystem
Start Active Sync.- Specified by:
startActiveSyncPolling
in interfaceUnderFileSystem
- Parameters:
txId
- the transaction id to start receiving event- Returns:
- true if active sync started
- Throws:
java.io.IOException
-
stopActiveSyncPolling
public boolean stopActiveSyncPolling()
Description copied from interface:UnderFileSystem
Stop Active Sync.- Specified by:
stopActiveSyncPolling
in interfaceUnderFileSystem
- Returns:
- true if active sync stopped
-
startSync
public void startSync(AlluxioURI uri)
Description copied from interface:UnderFileSystem
Add Sync Point.- Specified by:
startSync
in interfaceUnderFileSystem
- Parameters:
uri
- ufs uri to start
-
stopSync
public void stopSync(AlluxioURI uri)
Description copied from interface:UnderFileSystem
Stop Sync Point.- Specified by:
stopSync
in interfaceUnderFileSystem
- Parameters:
uri
- ufs uri to stop
-
validatePath
protected static java.lang.String validatePath(java.lang.String path)
Clean the path by creating a URI and turning it back to a string.- Parameters:
path
- the path to validate- Returns:
- validated path
-
-