Class ObjectUnderFileSystem
- java.lang.Object
-
- alluxio.underfs.BaseUnderFileSystem
-
- alluxio.underfs.ObjectUnderFileSystem
-
- All Implemented Interfaces:
UnderFileSystem
,java.io.Closeable
,java.lang.AutoCloseable
@ThreadSafe public abstract class ObjectUnderFileSystem extends BaseUnderFileSystem
An object based abstractUnderFileSystem
. Object Stores implementing theUnderFileSystem
interface should derive from this class.
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description protected class
ObjectUnderFileSystem.DeleteBuffer
Object keys added to aObjectUnderFileSystem.DeleteBuffer
will be deleted in batches.static interface
ObjectUnderFileSystem.ObjectListingChunk
A chunk of listing results.class
ObjectUnderFileSystem.ObjectPermissions
Permissions in object UFS.protected class
ObjectUnderFileSystem.ObjectStatus
Information about a single object in object UFS.protected static interface
ObjectUnderFileSystem.ObjectStoreOperation<T>
Represents an object store operation.protected class
ObjectUnderFileSystem.OperationBuffer<T>
Operations added to this buffer are performed concurrently.protected class
ObjectUnderFileSystem.RenameBuffer
File paths added to aObjectUnderFileSystem.RenameBuffer
will be renamed concurrently.-
Nested classes/interfaces inherited from interface alluxio.underfs.UnderFileSystem
UnderFileSystem.Factory, UnderFileSystem.SpaceType
-
-
Field Summary
Fields Modifier and Type Field Description protected java.util.concurrent.ExecutorService
mExecutorService
Executor service used for parallel UFS operations such as bulk deletes.protected java.util.function.Supplier<java.lang.String>
mRootKeySupplier
The root key of an object fs.protected static java.lang.String
PATH_SEPARATOR
Value used to indicate nested structure.protected static char
PATH_SEPARATOR_CHAR
Value used to indicate nested structure.-
Fields inherited from class alluxio.underfs.BaseUnderFileSystem
EMPTY_ACL, mUfsConf, mUri
-
-
Constructor Summary
Constructors Modifier Constructor Description protected
ObjectUnderFileSystem(AlluxioURI uri, UnderFileSystemConfiguration ufsConf)
Constructs anObjectUnderFileSystem
.
-
Method Summary
All Methods Instance Methods Abstract Methods Concrete Methods Modifier and Type Method Description void
cleanup()
Cleans up the under file system.void
close()
void
connectFromMaster(java.lang.String hostname)
Takes any necessary actions required to establish a connection to the under file system from the given master host e.g.void
connectFromWorker(java.lang.String hostname)
Takes any necessary actions required to establish a connection to the under file system from the given worker host e.g.protected java.lang.String
convertToFolderName(java.lang.String key)
Appends the directory suffix to the key.protected abstract boolean
copyObject(java.lang.String src, java.lang.String dst)
Copies an object to another key.java.io.OutputStream
create(java.lang.String path, CreateOptions options)
Creates a file in the under file system with the specifiedCreateOptions
.abstract boolean
createEmptyObject(java.lang.String key)
Creates a zero-byte object used to encode a directory.java.io.OutputStream
createNonexistingFile(java.lang.String path)
Creates a file in the under file system with the indicated name.java.io.OutputStream
createNonexistingFile(java.lang.String path, CreateOptions options)
Creates a file in the under file system with the specifiedCreateOptions
.protected abstract java.io.OutputStream
createObject(java.lang.String key)
Creates anOutputStream
for object uploads.boolean
deleteDirectory(java.lang.String path, DeleteOptions options)
Deletes a directory from the under file system with the indicated name.boolean
deleteExistingDirectory(java.lang.String path)
Deletes a directory from the under file system.boolean
deleteExistingDirectory(java.lang.String path, DeleteOptions options)
Deletes a directory from the under file system with the indicated name.boolean
deleteExistingFile(java.lang.String path)
Deletes a file from the under file system with the indicated name.boolean
deleteFile(java.lang.String path)
Deletes a file from the under file system with the indicated name.protected abstract boolean
deleteObject(java.lang.String key)
Internal function to delete a key.protected java.util.List<java.lang.String>
deleteObjects(java.util.List<java.lang.String> keys)
Internal function to delete a list of keys.long
getBlockSizeByte(java.lang.String path)
Gets the block size in bytes.protected java.lang.String
getChildName(java.lang.String child, java.lang.String parent)
Gets the child name based on the parent name.UfsDirectoryStatus
getDirectoryStatus(java.lang.String path)
Gets the directory status.UfsDirectoryStatus
getExistingDirectoryStatus(java.lang.String path)
Gets the directory status.UfsFileStatus
getExistingFileStatus(java.lang.String path)
Gets the file status.UfsStatus
getExistingStatus(java.lang.String path)
Gets the file or directory status.java.util.List<java.lang.String>
getFileLocations(java.lang.String path)
Gets the list of locations of the indicated path.java.util.List<java.lang.String>
getFileLocations(java.lang.String path, FileLocationOptions options)
Gets the list of locations of the indicated path given options.UfsFileStatus
getFileStatus(java.lang.String path)
Gets the file status.protected abstract java.lang.String
getFolderSuffix()
Get suffix used to encode a directory.protected int
getListingChunkLength(AlluxioConfiguration conf)
The number of items to query in a single listing chunk.protected int
getListingChunkLengthMax()
Maximum number of items in a single listing chunk supported by the under store.protected abstract ObjectUnderFileSystem.ObjectListingChunk
getObjectListingChunk(java.lang.String key, boolean recursive)
Gets a (partial) object listing result for the given key.protected ObjectUnderFileSystem.ObjectListingChunk
getObjectListingChunkForPath(java.lang.String path, boolean recursive)
Gets a (partial) object listing for the given path.protected abstract ObjectUnderFileSystem.ObjectStatus
getObjectStatus(java.lang.String key)
Get metadata information about object.protected java.lang.String
getParentPath(java.lang.String path)
Get parent path.protected abstract ObjectUnderFileSystem.ObjectPermissions
getPermissions()
Permissions for the mounted bucket.protected abstract java.lang.String
getRootKey()
Get full path of root in object store.long
getSpace(java.lang.String path, UnderFileSystem.SpaceType type)
Queries the under file system about the space of the indicated path (e.g., space left, space used and etc).UfsStatus
getStatus(java.lang.String path)
Gets the file or directory status.boolean
isDirectory(java.lang.String path)
Checks if a directory exists in under file system.boolean
isExistingDirectory(java.lang.String path)
Checks if a directory exists in under file system.boolean
isFile(java.lang.String path)
Checks if a file exists in under file system.boolean
isObjectStorage()
protected boolean
isRoot(java.lang.String path)
Checks if the path is the root.protected UfsStatus[]
listInternal(java.lang.String path, ListOptions options)
Lists the files in the given path, the paths will be their logical names and not contain the folder suffix.UfsStatus[]
listStatus(java.lang.String path)
Returns an array of statuses of the files and directories in the directory denoted by this abstract pathname.UfsStatus[]
listStatus(java.lang.String path, ListOptions options)
Returns an array of statuses of the files and directories in the directory denoted by this abstract pathname, with options.boolean
mkdirs(java.lang.String path, MkdirsOptions options)
Creates the directory named by this abstract pathname, with specifiedMkdirsOptions
.protected boolean
mkdirsInternal(java.lang.String key)
Creates a directory flagged file with the key and folder suffix.java.io.InputStream
open(java.lang.String path, OpenOptions options)
Opens anInputStream
for a file in under filesystem at the indicated path.java.io.InputStream
openExistingFile(java.lang.String path)
Opens anInputStream
for a file in under filesystem at the indicated path.java.io.InputStream
openExistingFile(java.lang.String path, OpenOptions options)
Opens anInputStream
for a file in under filesystem at the indicated path.protected abstract java.io.InputStream
openObject(java.lang.String key, OpenOptions options, RetryPolicy retryPolicy)
Internal function to open an input stream to an object.protected boolean
parentExists(java.lang.String path)
Treating the object store as a file system, checks if the parent directory exists.boolean
renameDirectory(java.lang.String src, java.lang.String dst)
Renames a directory fromsrc
todst
in under file system.boolean
renameFile(java.lang.String src, java.lang.String dst)
Renames a file fromsrc
todst
in under file system.boolean
renameRenamableDirectory(java.lang.String src, java.lang.String dst)
Renames a directory fromsrc
todst
in under file system.boolean
renameRenamableFile(java.lang.String src, java.lang.String dst)
Renames a file fromsrc
todst
in under file system.protected <T> T
retryOnException(ObjectUnderFileSystem.ObjectStoreOperation<T> op, java.util.function.Supplier<java.lang.String> description)
Retries the given object store operation when it throws exceptions to resolve eventual consistency issue.java.lang.String
stripPrefixIfPresent(java.lang.String path)
Strips the bucket prefix or the preceding path separator from the path if it is present.boolean
supportsFlush()
Whether this type of UFS supports flush.-
Methods inherited from class alluxio.underfs.BaseUnderFileSystem
create, deleteDirectory, exists, getAclPair, getActiveSyncInfo, getConfiguration, getFingerprint, getOperationMode, getParsedFingerprint, getPhysicalStores, isSeekable, mkdirs, open, resolveUri, setAclEntries, startActiveSyncPolling, startSync, stopActiveSyncPolling, stopSync, supportsActiveSync, validatePath
-
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
-
Methods inherited from interface alluxio.underfs.UnderFileSystem
getUnderFSType, listStatuses, openPositionRead, setMode, setOwner
-
-
-
-
Field Detail
-
PATH_SEPARATOR_CHAR
protected static final char PATH_SEPARATOR_CHAR
Value used to indicate nested structure.- See Also:
- Constant Field Values
-
PATH_SEPARATOR
protected static final java.lang.String PATH_SEPARATOR
Value used to indicate nested structure. This is a string representation ofPATH_SEPARATOR_CHAR
.
-
mExecutorService
protected java.util.concurrent.ExecutorService mExecutorService
Executor service used for parallel UFS operations such as bulk deletes.
-
mRootKeySupplier
protected final java.util.function.Supplier<java.lang.String> mRootKeySupplier
The root key of an object fs.
-
-
Constructor Detail
-
ObjectUnderFileSystem
protected ObjectUnderFileSystem(AlluxioURI uri, UnderFileSystemConfiguration ufsConf)
Constructs anObjectUnderFileSystem
.- Parameters:
uri
- theAlluxioURI
used to create this ufsufsConf
- UFS configuration
-
-
Method Detail
-
cleanup
public void cleanup() throws java.io.IOException
Description copied from interface:UnderFileSystem
Cleans up the under file system. If any data or files are created and not completed/aborted correctly in normal ways, they should be cleaned in this method.- Throws:
java.io.IOException
-
close
public void close() throws java.io.IOException
- Throws:
java.io.IOException
-
connectFromMaster
public void connectFromMaster(java.lang.String hostname)
Description copied from interface:UnderFileSystem
Takes any necessary actions required to establish a connection to the under file system from the given master host e.g. logging inDepending on the implementation this may be a no-op
- Parameters:
hostname
- the host that wants to connect to the under file system
-
connectFromWorker
public void connectFromWorker(java.lang.String hostname)
Description copied from interface:UnderFileSystem
Takes any necessary actions required to establish a connection to the under file system from the given worker host e.g. logging inDepending on the implementation this may be a no-op
- Parameters:
hostname
- the host that wants to connect to the under file system
-
create
public java.io.OutputStream create(java.lang.String path, CreateOptions options) throws java.io.IOException
Description copied from interface:UnderFileSystem
Creates a file in the under file system with the specifiedCreateOptions
. Implementations should make sure that the path under creation appears in listings only after a successful close and that contents are written in its entirety or not at all.- Parameters:
path
- the file nameoptions
- the options for create- Returns:
- A
OutputStream
object - Throws:
java.io.IOException
-
createNonexistingFile
public java.io.OutputStream createNonexistingFile(java.lang.String path) throws java.io.IOException
Description copied from interface:UnderFileSystem
Creates a file in the under file system with the indicated name. Similar toUnderFileSystem.create(String)
but deals with the delete-then-create eventual consistency issue.- Parameters:
path
- the file name- Returns:
- A
OutputStream
object - Throws:
java.io.IOException
-
createNonexistingFile
public java.io.OutputStream createNonexistingFile(java.lang.String path, CreateOptions options) throws java.io.IOException
Description copied from interface:UnderFileSystem
Creates a file in the under file system with the specifiedCreateOptions
. Similar toUnderFileSystem.create(String, CreateOptions)
but deals with the delete-then-create eventual consistency issue.- Parameters:
path
- the file nameoptions
- the options for create- Returns:
- A
OutputStream
object - Throws:
java.io.IOException
-
deleteFile
public boolean deleteFile(java.lang.String path) throws java.io.IOException
Description copied from interface:UnderFileSystem
Deletes a file from the under file system with the indicated name.- Parameters:
path
- of the file to delete- Returns:
- true if file was found and deleted, false otherwise
- Throws:
java.io.IOException
-
deleteExistingFile
public boolean deleteExistingFile(java.lang.String path) throws java.io.IOException
Description copied from interface:UnderFileSystem
Deletes a file from the under file system with the indicated name. Similar toUnderFileSystem.deleteFile(String)
but deals with the create-then-delete eventual consistency issue.- Parameters:
path
- of the file to delete- Returns:
- true if file was found and deleted, false otherwise
- Throws:
java.io.IOException
-
deleteDirectory
public boolean deleteDirectory(java.lang.String path, DeleteOptions options) throws java.io.IOException
Description copied from interface:UnderFileSystem
Deletes a directory from the under file system with the indicated name.- Parameters:
path
- of the directory to deleteoptions
- for directory delete semantics- Returns:
- true if directory was found and deleted, false otherwise
- Throws:
java.io.IOException
-
deleteExistingDirectory
public boolean deleteExistingDirectory(java.lang.String path) throws java.io.IOException
Description copied from interface:UnderFileSystem
Deletes a directory from the under file system. Similar toUnderFileSystem.deleteDirectory(String)
but deals with the create-delete eventual consistency issue.- Parameters:
path
- of the directory to delete- Returns:
- true if directory was found and deleted, false otherwise
- Throws:
java.io.IOException
-
deleteExistingDirectory
public boolean deleteExistingDirectory(java.lang.String path, DeleteOptions options) throws java.io.IOException
Description copied from interface:UnderFileSystem
Deletes a directory from the under file system with the indicated name. Similar toUnderFileSystem.deleteDirectory(String, DeleteOptions)
but deals with the create-then-delete eventual consistency issue.- Parameters:
path
- of the directory to deleteoptions
- for directory delete semantics- Returns:
- true if directory was found and deleted, false otherwise
- Throws:
java.io.IOException
-
getBlockSizeByte
public long getBlockSizeByte(java.lang.String path) throws java.io.IOException
Gets the block size in bytes. This method defaults to the default user block size in Alluxio.- Parameters:
path
- the file name- Returns:
- the default Alluxio user block size
- Throws:
java.io.IOException
-
getDirectoryStatus
public UfsDirectoryStatus getDirectoryStatus(java.lang.String path) throws java.io.IOException
Description copied from interface:UnderFileSystem
Gets the directory status. The caller must already know the path is a directory. This method will throw an exception if the path exists, but is a file.- Parameters:
path
- the path to the directory- Returns:
- the directory status
- Throws:
java.io.FileNotFoundException
- when the path does not existjava.io.IOException
-
getExistingDirectoryStatus
public UfsDirectoryStatus getExistingDirectoryStatus(java.lang.String path) throws java.io.IOException
Description copied from interface:UnderFileSystem
Gets the directory status. Similar toUnderFileSystem.getDirectoryStatus(String)
but deals with the write-then-get-status eventual consistency issue.- Parameters:
path
- the path to the directory- Returns:
- the directory status
- Throws:
java.io.IOException
-
getFileLocations
@Nullable public java.util.List<java.lang.String> getFileLocations(java.lang.String path) throws java.io.IOException
Description copied from interface:UnderFileSystem
Gets the list of locations of the indicated path.- Parameters:
path
- the file name- Returns:
- The list of locations
- Throws:
java.io.IOException
-
getFileLocations
@Nullable public java.util.List<java.lang.String> getFileLocations(java.lang.String path, FileLocationOptions options) throws java.io.IOException
Description copied from interface:UnderFileSystem
Gets the list of locations of the indicated path given options.- Parameters:
path
- the file nameoptions
- method options- Returns:
- The list of locations
- Throws:
java.io.IOException
-
getSpace
public long getSpace(java.lang.String path, UnderFileSystem.SpaceType type) throws java.io.IOException
Description copied from interface:UnderFileSystem
Queries the under file system about the space of the indicated path (e.g., space left, space used and etc).- Parameters:
path
- the path to querytype
- the type of queries- Returns:
- The space in bytes
- Throws:
java.io.IOException
-
getFileStatus
public UfsFileStatus getFileStatus(java.lang.String path) throws java.io.IOException
Description copied from interface:UnderFileSystem
Gets the file status. The caller must already know the path is a file. This method will throw an exception if the path exists, but is a directory.- Parameters:
path
- the path to the file- Returns:
- the file status
- Throws:
java.io.FileNotFoundException
- when the path does not existjava.io.IOException
-
getExistingFileStatus
public UfsFileStatus getExistingFileStatus(java.lang.String path) throws java.io.IOException
Description copied from interface:UnderFileSystem
Gets the file status. Similar toUnderFileSystem.getFileStatus(String)
but deals with the write-then-get-status eventual consistency issue.- Parameters:
path
- the path to the file- Returns:
- the file status
- Throws:
java.io.IOException
-
getStatus
public UfsStatus getStatus(java.lang.String path) throws java.io.IOException
Description copied from interface:UnderFileSystem
Gets the file or directory status. The caller does not need to know if the path is a file or directory. This method will determine the path type, and will return the appropriate status.- Parameters:
path
- the path to get the status- Returns:
- the file or directory status
- Throws:
java.io.FileNotFoundException
- when the path does not existjava.io.IOException
-
getExistingStatus
public UfsStatus getExistingStatus(java.lang.String path) throws java.io.IOException
Description copied from interface:UnderFileSystem
Gets the file or directory status. Similar toUnderFileSystem.getStatus(String)
but deals with the write-then-get-status eventual consistency issue.- Parameters:
path
- the path to get the status- Returns:
- the file or directory status
- Throws:
java.io.IOException
-
isDirectory
public boolean isDirectory(java.lang.String path) throws java.io.IOException
Description copied from interface:UnderFileSystem
Checks if a directory exists in under file system.- Parameters:
path
- the absolute directory path- Returns:
- true if the path exists and is a directory, false otherwise
- Throws:
java.io.IOException
-
isExistingDirectory
public boolean isExistingDirectory(java.lang.String path) throws java.io.IOException
Description copied from interface:UnderFileSystem
Checks if a directory exists in under file system. Similar toUnderFileSystem.isDirectory(String)
but deals with the write-then-list eventual consistency issue.- Parameters:
path
- the absolute directory path- Returns:
- true if the path exists and is a directory, false otherwise
- Throws:
java.io.IOException
-
isFile
public boolean isFile(java.lang.String path) throws java.io.IOException
Description copied from interface:UnderFileSystem
Checks if a file exists in under file system.- Parameters:
path
- the absolute file path- Returns:
- true if the path exists and is a file, false otherwise
- Throws:
java.io.IOException
-
isObjectStorage
public boolean isObjectStorage()
- Specified by:
isObjectStorage
in interfaceUnderFileSystem
- Overrides:
isObjectStorage
in classBaseUnderFileSystem
- Returns:
- true if under storage is an object store, false otherwise
-
listStatus
public UfsStatus[] listStatus(java.lang.String path) throws java.io.IOException
Description copied from interface:UnderFileSystem
Returns an array of statuses of the files and directories in the directory denoted by this abstract pathname.If this abstract pathname does not denote a directory, then this method returns
null
. Otherwise an array of statuses is returned, one for each file or directory in the directory. Names denoting the directory itself and the directory's parent directory are not included in the result. Each string is a file name rather than a complete path.There is no guarantee that the name strings in the resulting array will appear in any specific order; they are not, in particular, guaranteed to appear in alphabetical order.
- Parameters:
path
- the abstract pathname to list- Returns:
- An array with the statuses of the files and directories in the directory denoted by
this abstract pathname. The array will be empty if the directory is empty. Returns
null
if this abstract pathname does not denote a directory. - Throws:
java.io.IOException
-
listStatus
public UfsStatus[] listStatus(java.lang.String path, ListOptions options) throws java.io.IOException
Description copied from interface:UnderFileSystem
Returns an array of statuses of the files and directories in the directory denoted by this abstract pathname, with options.If this abstract pathname does not denote a directory, then this method returns
null
. Otherwise an array of statuses is returned, one for each file or directory. Names denoting the directory itself and the directory's parent directory are not included in the result. Each string is a path relative to the given directory.There is no guarantee that the name strings in the resulting array will appear in any specific order; they are not, in particular, guaranteed to appear in alphabetical order.
- Specified by:
listStatus
in interfaceUnderFileSystem
- Overrides:
listStatus
in classBaseUnderFileSystem
- Parameters:
path
- the abstract pathname to listoptions
- for list directory- Returns:
- An array of statuses naming the files and directories in the directory denoted by this
abstract pathname. The array will be empty if the directory is empty. Returns
null
if this abstract pathname does not denote a directory. - Throws:
java.io.IOException
-
mkdirs
public boolean mkdirs(java.lang.String path, MkdirsOptions options) throws java.io.IOException
Description copied from interface:UnderFileSystem
Creates the directory named by this abstract pathname, with specifiedMkdirsOptions
. If the folder already exists, the method returns false.- Parameters:
path
- the folder to createoptions
- the options for mkdirs- Returns:
true
if and only if the directory was created;false
otherwise- Throws:
java.io.IOException
-
open
public java.io.InputStream open(java.lang.String path, OpenOptions options) throws java.io.IOException
Description copied from interface:UnderFileSystem
Opens anInputStream
for a file in under filesystem at the indicated path.- Parameters:
path
- the file nameoptions
- to open input stream- Returns:
- The
InputStream
object - Throws:
java.io.IOException
-
openExistingFile
public java.io.InputStream openExistingFile(java.lang.String path) throws java.io.IOException
Description copied from interface:UnderFileSystem
Opens anInputStream
for a file in under filesystem at the indicated path. Similar toUnderFileSystem.open(String)
but deals with the write-then-read eventual consistency issue.- Parameters:
path
- the file name- Returns:
- The
InputStream
object - Throws:
java.io.IOException
-
openExistingFile
public java.io.InputStream openExistingFile(java.lang.String path, OpenOptions options) throws java.io.IOException
Description copied from interface:UnderFileSystem
Opens anInputStream
for a file in under filesystem at the indicated path. Similar toUnderFileSystem.open(String, OpenOptions)
but deals with the write-then-read eventual consistency issue.- Parameters:
path
- the file nameoptions
- to open input stream- Returns:
- The
InputStream
object - Throws:
java.io.IOException
-
renameDirectory
public boolean renameDirectory(java.lang.String src, java.lang.String dst) throws java.io.IOException
Description copied from interface:UnderFileSystem
Renames a directory fromsrc
todst
in under file system.- Parameters:
src
- the source directory pathdst
- the destination directory path- Returns:
- true if succeed, false otherwise
- Throws:
java.io.IOException
-
renameRenamableDirectory
public boolean renameRenamableDirectory(java.lang.String src, java.lang.String dst) throws java.io.IOException
Description copied from interface:UnderFileSystem
Renames a directory fromsrc
todst
in under file system. Similar toUnderFileSystem.renameDirectory(String, String)
but deals with the write-src-then-rename and delete-dst-then-rename eventual consistency issue.- Parameters:
src
- the source directory pathdst
- the destination directory path- Returns:
- true if succeed, false otherwise
- Throws:
java.io.IOException
-
renameFile
public boolean renameFile(java.lang.String src, java.lang.String dst) throws java.io.IOException
Description copied from interface:UnderFileSystem
Renames a file fromsrc
todst
in under file system.- Parameters:
src
- the source file pathdst
- the destination file path- Returns:
- true if succeed, false otherwise
- Throws:
java.io.IOException
-
renameRenamableFile
public boolean renameRenamableFile(java.lang.String src, java.lang.String dst) throws java.io.IOException
Description copied from interface:UnderFileSystem
Renames a file fromsrc
todst
in under file system. Similar toUnderFileSystem.renameFile(String, String)
but deals with the write-src-then-rename and delete-dst-then-rename eventual consistency issue.- Parameters:
src
- the source file pathdst
- the destination file path- Returns:
- true if succeed, false otherwise
- Throws:
java.io.IOException
-
supportsFlush
public boolean supportsFlush() throws java.io.IOException
Description copied from interface:UnderFileSystem
Whether this type of UFS supports flush.- Returns:
- true if this type of UFS supports flush, false otherwise
- Throws:
java.io.IOException
-
createEmptyObject
public abstract boolean createEmptyObject(java.lang.String key)
Creates a zero-byte object used to encode a directory.- Parameters:
key
- the key to create- Returns:
- true if the operation was successful
-
createObject
protected abstract java.io.OutputStream createObject(java.lang.String key) throws java.io.IOException
Creates anOutputStream
for object uploads.- Parameters:
key
- ufs key including scheme and bucket- Returns:
- new OutputStream
- Throws:
java.io.IOException
-
convertToFolderName
protected java.lang.String convertToFolderName(java.lang.String key)
Appends the directory suffix to the key.- Parameters:
key
- the key to convert- Returns:
- key as a directory path
-
copyObject
protected abstract boolean copyObject(java.lang.String src, java.lang.String dst) throws java.io.IOException
Copies an object to another key.- Parameters:
src
- the source key to copydst
- the destination key to copy to- Returns:
- true if the operation was successful, false otherwise
- Throws:
java.io.IOException
-
deleteObject
protected abstract boolean deleteObject(java.lang.String key) throws java.io.IOException
Internal function to delete a key.- Parameters:
key
- the key to delete- Returns:
- true if successful, false if an exception is thrown
- Throws:
java.io.IOException
-
deleteObjects
protected java.util.List<java.lang.String> deleteObjects(java.util.List<java.lang.String> keys) throws java.io.IOException
Internal function to delete a list of keys.- Parameters:
keys
- the list of keys to delete- Returns:
- list of successfully deleted keys
- Throws:
java.io.IOException
-
getPermissions
protected abstract ObjectUnderFileSystem.ObjectPermissions getPermissions()
Permissions for the mounted bucket.- Returns:
- permissions
-
getListingChunkLengthMax
protected int getListingChunkLengthMax()
Maximum number of items in a single listing chunk supported by the under store.- Returns:
- the maximum length for a single listing query
-
getListingChunkLength
protected int getListingChunkLength(AlluxioConfiguration conf)
The number of items to query in a single listing chunk.- Returns:
- length of each list request
-
getObjectStatus
@Nullable protected abstract ObjectUnderFileSystem.ObjectStatus getObjectStatus(java.lang.String key) throws java.io.IOException
Get metadata information about object. Implementations should process the key as is, which may be a file or a directory key.- Parameters:
key
- ufs key to get metadata for- Returns:
ObjectUnderFileSystem.ObjectStatus
if key exists and successful, otherwise null- Throws:
java.io.IOException
-
getParentPath
@Nullable protected java.lang.String getParentPath(java.lang.String path)
Get parent path.- Parameters:
path
- ufs path including scheme and bucket- Returns:
- the parent path, or null if the parent does not exist
-
isRoot
protected boolean isRoot(java.lang.String path)
Checks if the path is the root. This method supports full path (e.g. s3://bucket_name/dir) and stripped path (e.g. /dir).- Parameters:
path
- ufs path including scheme and bucket- Returns:
- true if the path is the root, false otherwise
-
getChildName
protected java.lang.String getChildName(java.lang.String child, java.lang.String parent) throws java.io.IOException
Gets the child name based on the parent name.- Parameters:
child
- the key of the childparent
- the key of the parent- Returns:
- the child key with the parent prefix removed
- Throws:
java.io.IOException
-
getFolderSuffix
protected abstract java.lang.String getFolderSuffix()
Get suffix used to encode a directory.- Returns:
- folder suffix
-
getObjectListingChunk
@Nullable protected abstract ObjectUnderFileSystem.ObjectListingChunk getObjectListingChunk(java.lang.String key, boolean recursive) throws java.io.IOException
Gets a (partial) object listing result for the given key.- Parameters:
key
- pseudo-directory key excluding header and bucketrecursive
- whether to request immediate children only, or all descendants- Returns:
- chunked object listing, or null if key is not found
- Throws:
java.io.IOException
-
getObjectListingChunkForPath
@Nullable protected ObjectUnderFileSystem.ObjectListingChunk getObjectListingChunkForPath(java.lang.String path, boolean recursive) throws java.io.IOException
Gets a (partial) object listing for the given path.- Parameters:
path
- of pseudo-directoryrecursive
- whether to request immediate children only, or all descendants- Returns:
- chunked object listing, or null if the path does not exist as a pseudo-directory
- Throws:
java.io.IOException
-
getRootKey
protected abstract java.lang.String getRootKey()
Get full path of root in object store.- Returns:
- full path including scheme and bucket
-
listInternal
@Nullable protected UfsStatus[] listInternal(java.lang.String path, ListOptions options) throws java.io.IOException
Lists the files in the given path, the paths will be their logical names and not contain the folder suffix. Note that, the list results are unsorted.- Parameters:
path
- the key to listoptions
- for listing- Returns:
- an array of the file and folder names in this directory
- Throws:
java.io.IOException
-
mkdirsInternal
protected boolean mkdirsInternal(java.lang.String key)
Creates a directory flagged file with the key and folder suffix.- Parameters:
key
- the key to create a folder- Returns:
- true if the operation was successful, false otherwise
-
openObject
protected abstract java.io.InputStream openObject(java.lang.String key, OpenOptions options, RetryPolicy retryPolicy) throws java.io.IOException
Internal function to open an input stream to an object.- Parameters:
key
- the key to openoptions
- the open optionsretryPolicy
- the retry policy of the opened stream to solve eventual consistency issue- Returns:
- an
InputStream
to read from key - Throws:
java.io.IOException
-
parentExists
protected boolean parentExists(java.lang.String path) throws java.io.IOException
Treating the object store as a file system, checks if the parent directory exists.- Parameters:
path
- the path to check- Returns:
- true if the parent exists or if the path is root, false otherwise
- Throws:
java.io.IOException
-
stripPrefixIfPresent
public java.lang.String stripPrefixIfPresent(java.lang.String path)
Strips the bucket prefix or the preceding path separator from the path if it is present. For example, for input path ufs://my-bucket-name/my-path/file, the output would be my-path/file. If path is an absolute path like /my-path/file, the output would be my-path/file. This method will leave keys without a prefix unaltered, ie. my-path/file returns my-path/file.- Parameters:
path
- the path to strip- Returns:
- the path without the bucket prefix
-
retryOnException
protected <T> T retryOnException(ObjectUnderFileSystem.ObjectStoreOperation<T> op, java.util.function.Supplier<java.lang.String> description) throws java.io.IOException
Retries the given object store operation when it throws exceptions to resolve eventual consistency issue.- Parameters:
op
- the object store operation to retrydescription
- the description regarding the operation- Returns:
- the operation result if operation succeed
- Throws:
java.io.IOException
-
-