Package alluxio.underfs

Class ObjectUnderFileSystem

    • Field Detail

      • PATH_SEPARATOR_CHAR

        protected static final char PATH_SEPARATOR_CHAR
        Value used to indicate nested structure.
        See Also:
        Constant Field Values

      • PATH_SEPARATOR

        protected static final java.lang.String PATH_SEPARATOR
        Value used to indicate nested structure. This is a string representation of PATH_SEPARATOR_CHAR.

      • mExecutorService

        protected java.util.concurrent.ExecutorService mExecutorService
        Executor service used for parallel UFS operations such as bulk deletes.

      • mRootKeySupplier

        protected final java.util.function.Supplier<java.lang.String> mRootKeySupplier
        The root key of an object fs.

    • Method Detail

      • cleanup

        public void cleanup()
             throws java.io.IOException
        Description copied from interface: UnderFileSystem
        Cleans up the under file system. If any data or files are created and not completed/aborted correctly in normal ways, they should be cleaned in this method.
        Throws:
        java.io.IOException

      • close

        public void close()
           throws java.io.IOException
        Specified by:
        close in interface java.lang.AutoCloseable
        Specified by:
        close in interface java.io.Closeable
        Overrides:
        close in class BaseUnderFileSystem
        Throws:
        java.io.IOException

      • connectFromMaster

        public void connectFromMaster​(java.lang.String hostname)
        Description copied from interface: UnderFileSystem
        Takes any necessary actions required to establish a connection to the under file system from the given master host e.g. logging in

        Depending on the implementation this may be a no-op

        Parameters:
        hostname - the host that wants to connect to the under file system

      • connectFromWorker

        public void connectFromWorker​(java.lang.String hostname)
        Description copied from interface: UnderFileSystem
        Takes any necessary actions required to establish a connection to the under file system from the given worker host e.g. logging in

        Depending on the implementation this may be a no-op

        Parameters:
        hostname - the host that wants to connect to the under file system

      • create

        public java.io.OutputStream create​(java.lang.String path,
                                   CreateOptions options)
                            throws java.io.IOException
        Description copied from interface: UnderFileSystem
        Creates a file in the under file system with the specified CreateOptions. Implementations should make sure that the path under creation appears in listings only after a successful close and that contents are written in its entirety or not at all.
        Parameters:
        path - the file name
        options - the options for create
        Returns:
        A OutputStream object
        Throws:
        java.io.IOException

      • createNonexistingFile

        public java.io.OutputStream createNonexistingFile​(java.lang.String path)
                                           throws java.io.IOException
        Description copied from interface: UnderFileSystem
        Creates a file in the under file system with the indicated name. Similar to UnderFileSystem.create(String) but deals with the delete-then-create eventual consistency issue.
        Parameters:
        path - the file name
        Returns:
        A OutputStream object
        Throws:
        java.io.IOException

      • createNonexistingFile

        public java.io.OutputStream createNonexistingFile​(java.lang.String path,
                                                  CreateOptions options)
                                           throws java.io.IOException
        Description copied from interface: UnderFileSystem
        Creates a file in the under file system with the specified CreateOptions. Similar to UnderFileSystem.create(String, CreateOptions) but deals with the delete-then-create eventual consistency issue.
        Parameters:
        path - the file name
        options - the options for create
        Returns:
        A OutputStream object
        Throws:
        java.io.IOException

      • deleteFile

        public boolean deleteFile​(java.lang.String path)
                   throws java.io.IOException
        Description copied from interface: UnderFileSystem
        Deletes a file from the under file system with the indicated name.
        Parameters:
        path - of the file to delete
        Returns:
        true if file was found and deleted, false otherwise
        Throws:
        java.io.IOException

      • deleteExistingFile

        public boolean deleteExistingFile​(java.lang.String path)
                           throws java.io.IOException
        Description copied from interface: UnderFileSystem
        Deletes a file from the under file system with the indicated name. Similar to UnderFileSystem.deleteFile(String) but deals with the create-then-delete eventual consistency issue.
        Parameters:
        path - of the file to delete
        Returns:
        true if file was found and deleted, false otherwise
        Throws:
        java.io.IOException

      • deleteDirectory

        public boolean deleteDirectory​(java.lang.String path,
                               DeleteOptions options)
                        throws java.io.IOException
        Description copied from interface: UnderFileSystem
        Deletes a directory from the under file system with the indicated name.
        Parameters:
        path - of the directory to delete
        options - for directory delete semantics
        Returns:
        true if directory was found and deleted, false otherwise
        Throws:
        java.io.IOException

      • deleteExistingDirectory

        public boolean deleteExistingDirectory​(java.lang.String path)
                                throws java.io.IOException
        Description copied from interface: UnderFileSystem
        Deletes a directory from the under file system. Similar to UnderFileSystem.deleteDirectory(String) but deals with the create-delete eventual consistency issue.
        Parameters:
        path - of the directory to delete
        Returns:
        true if directory was found and deleted, false otherwise
        Throws:
        java.io.IOException

      • deleteExistingDirectory

        public boolean deleteExistingDirectory​(java.lang.String path,
                                       DeleteOptions options)
                                throws java.io.IOException
        Description copied from interface: UnderFileSystem
        Deletes a directory from the under file system with the indicated name. Similar to UnderFileSystem.deleteDirectory(String, DeleteOptions) but deals with the create-then-delete eventual consistency issue.
        Parameters:
        path - of the directory to delete
        options - for directory delete semantics
        Returns:
        true if directory was found and deleted, false otherwise
        Throws:
        java.io.IOException

      • getBlockSizeByte

        public long getBlockSizeByte​(java.lang.String path)
                      throws java.io.IOException
        Gets the block size in bytes. This method defaults to the default user block size in Alluxio.
        Parameters:
        path - the file name
        Returns:
        the default Alluxio user block size
        Throws:
        java.io.IOException

      • getDirectoryStatus

        public UfsDirectoryStatus getDirectoryStatus​(java.lang.String path)
                                      throws java.io.IOException
        Description copied from interface: UnderFileSystem
        Gets the directory status. The caller must already know the path is a directory. This method will throw an exception if the path exists, but is a file.
        Parameters:
        path - the path to the directory
        Returns:
        the directory status
        Throws:
        java.io.FileNotFoundException - when the path does not exist
        java.io.IOException

      • getExistingDirectoryStatus

        public UfsDirectoryStatus getExistingDirectoryStatus​(java.lang.String path)
                                              throws java.io.IOException
        Description copied from interface: UnderFileSystem
        Gets the directory status. Similar to UnderFileSystem.getDirectoryStatus(String) but deals with the write-then-get-status eventual consistency issue.
        Parameters:
        path - the path to the directory
        Returns:
        the directory status
        Throws:
        java.io.IOException

      • getFileLocations

        @Nullable
public java.util.List<java.lang.String> getFileLocations​(java.lang.String path)
                                                  throws java.io.IOException
        Description copied from interface: UnderFileSystem
        Gets the list of locations of the indicated path.
        Parameters:
        path - the file name
        Returns:
        The list of locations
        Throws:
        java.io.IOException

      • getFileLocations

        @Nullable
public java.util.List<java.lang.String> getFileLocations​(java.lang.String path,
                                                         FileLocationOptions options)
                                                  throws java.io.IOException
        Description copied from interface: UnderFileSystem
        Gets the list of locations of the indicated path given options.
        Parameters:
        path - the file name
        options - method options
        Returns:
        The list of locations
        Throws:
        java.io.IOException

      • getSpace

        public long getSpace​(java.lang.String path,
                     UnderFileSystem.SpaceType type)
              throws java.io.IOException
        Description copied from interface: UnderFileSystem
        Queries the under file system about the space of the indicated path (e.g., space left, space used and etc).
        Parameters:
        path - the path to query
        type - the type of queries
        Returns:
        The space in bytes
        Throws:
        java.io.IOException

      • getFileStatus

        public UfsFileStatus getFileStatus​(java.lang.String path,
                                   GetStatusOptions options)
                            throws java.io.IOException
        Description copied from interface: UnderFileSystem
        Gets the file status. The caller must already know the path is a file. This method will throw an exception if the path exists, but is a directory.
        Parameters:
        path - the path to the file
        options - method options
        Returns:
        the file status
        Throws:
        java.io.FileNotFoundException - when the path does not exist
        java.io.IOException

      • getExistingFileStatus

        public UfsFileStatus getExistingFileStatus​(java.lang.String path)
                                    throws java.io.IOException
        Description copied from interface: UnderFileSystem
        Gets the file status. Similar to UnderFileSystem.getFileStatus(String) but deals with the write-then-get-status eventual consistency issue.
        Parameters:
        path - the path to the file
        Returns:
        the file status
        Throws:
        java.io.IOException

      • getStatus

        public UfsStatus getStatus​(java.lang.String path,
                           GetStatusOptions options)
                    throws java.io.IOException
        Description copied from interface: UnderFileSystem
        Gets the file or directory status. The caller does not need to know if the path is a file or directory. This method will determine the path type, and will return the appropriate status.
        Parameters:
        path - the path to get the status
        options - method options
        Returns:
        the file or directory status
        Throws:
        java.io.FileNotFoundException - when the path does not exist
        java.io.IOException

      • getExistingStatus

        public UfsStatus getExistingStatus​(java.lang.String path)
                            throws java.io.IOException
        Description copied from interface: UnderFileSystem
        Gets the file or directory status. Similar to UnderFileSystem.getStatus(String) but deals with the write-then-get-status eventual consistency issue.
        Parameters:
        path - the path to get the status
        Returns:
        the file or directory status
        Throws:
        java.io.IOException

      • isDirectory

        public boolean isDirectory​(java.lang.String path)
                    throws java.io.IOException
        Description copied from interface: UnderFileSystem
        Checks if a directory exists in under file system.
        Parameters:
        path - the absolute directory path
        Returns:
        true if the path exists and is a directory, false otherwise
        Throws:
        java.io.IOException

      • isExistingDirectory

        public boolean isExistingDirectory​(java.lang.String path)
                            throws java.io.IOException
        Description copied from interface: UnderFileSystem
        Checks if a directory exists in under file system. Similar to UnderFileSystem.isDirectory(String) but deals with the write-then-list eventual consistency issue.
        Parameters:
        path - the absolute directory path
        Returns:
        true if the path exists and is a directory, false otherwise
        Throws:
        java.io.IOException

      • isFile

        public boolean isFile​(java.lang.String path)
               throws java.io.IOException
        Description copied from interface: UnderFileSystem
        Checks if a file exists in under file system.
        Parameters:
        path - the absolute file path
        Returns:
        true if the path exists and is a file, false otherwise
        Throws:
        java.io.IOException

      • listStatus

        public UfsStatus[] listStatus​(java.lang.String path)
                       throws java.io.IOException
        Description copied from interface: UnderFileSystem
        Returns an array of statuses of the files and directories in the directory denoted by this abstract pathname.

        If this abstract pathname does not denote a directory, then this method returns null. Otherwise an array of statuses is returned, one for each file or directory in the directory. Names denoting the directory itself and the directory's parent directory are not included in the result. Each string is a file name rather than a complete path.

        There is no guarantee that the name strings in the resulting array will appear in any specific order; they are not, in particular, guaranteed to appear in alphabetical order.

        Parameters:
        path - the abstract pathname to list
        Returns:
        An array with the statuses of the files and directories in the directory denoted by this abstract pathname. The array will be empty if the directory is empty. Returns null if this abstract pathname does not denote a directory.
        Throws:
        java.io.IOException

      • listStatus

        public UfsStatus[] listStatus​(java.lang.String path,
                              ListOptions options)
                       throws java.io.IOException
        Description copied from interface: UnderFileSystem
        Returns an array of statuses of the files and directories in the directory denoted by this abstract pathname, with options.

        If this abstract pathname does not denote a directory, then this method returns null. Otherwise an array of statuses is returned, one for each file or directory. Names denoting the directory itself and the directory's parent directory are not included in the result. Each string is a path relative to the given directory.

        There is no guarantee that the name strings in the resulting array will appear in any specific order; they are not, in particular, guaranteed to appear in alphabetical order.

        Specified by:
        listStatus in interface UnderFileSystem
        Overrides:
        listStatus in class BaseUnderFileSystem
        Parameters:
        path - the abstract pathname to list
        options - for list directory
        Returns:
        An array of statuses naming the files and directories in the directory denoted by this abstract pathname. The array will be empty if the directory is empty. Returns null if this abstract pathname does not denote a directory.
        Throws:
        java.io.IOException

      • listStatusIterable

        @Nullable
public java.util.Iterator<UfsStatus> listStatusIterable​(java.lang.String path,
                                                        ListOptions options,
                                                        java.lang.String startAfter,
                                                        int batchSize)
                                                 throws java.io.IOException
        Description copied from interface: UnderFileSystem
        Lists the ufs statuses iteratively.
        Specified by:
        listStatusIterable in interface UnderFileSystem
        Overrides:
        listStatusIterable in class BaseUnderFileSystem
        Parameters:
        path - the abstract pathname to list
        options - for list directory
        startAfter - the start after token
        batchSize - the batch size
        Returns:
        An iterator of ufs status. Returns null if this abstract pathname does not denote a directory.
        Throws:
        java.io.IOException

      • mkdirs

        public boolean mkdirs​(java.lang.String path,
                      MkdirsOptions options)
               throws java.io.IOException
        Description copied from interface: UnderFileSystem
        Creates the directory named by this abstract pathname, with specified MkdirsOptions. If the folder already exists, the method returns false.
        Parameters:
        path - the folder to create
        options - the options for mkdirs
        Returns:
        true if and only if the directory was created; false otherwise
        Throws:
        java.io.IOException

      • open

        public java.io.InputStream open​(java.lang.String path,
                                OpenOptions options)
                         throws java.io.IOException
        Description copied from interface: UnderFileSystem
        Opens an InputStream for a file in under filesystem at the indicated path.
        Parameters:
        path - the file name
        options - to open input stream
        Returns:
        The InputStream object
        Throws:
        java.io.IOException

      • openExistingFile

        public java.io.InputStream openExistingFile​(java.lang.String path)
                                     throws java.io.IOException
        Description copied from interface: UnderFileSystem
        Opens an InputStream for a file in under filesystem at the indicated path. Similar to UnderFileSystem.open(String) but deals with the write-then-read eventual consistency issue.
        Parameters:
        path - the file name
        Returns:
        The InputStream object
        Throws:
        java.io.IOException

      • openExistingFile

        public java.io.InputStream openExistingFile​(java.lang.String path,
                                            OpenOptions options)
                                     throws java.io.IOException
        Description copied from interface: UnderFileSystem
        Opens an InputStream for a file in under filesystem at the indicated path. Similar to UnderFileSystem.open(String, OpenOptions) but deals with the write-then-read eventual consistency issue.
        Parameters:
        path - the file name
        options - to open input stream
        Returns:
        The InputStream object
        Throws:
        java.io.IOException

      • renameDirectory

        public boolean renameDirectory​(java.lang.String src,
                               java.lang.String dst)
                        throws java.io.IOException
        Description copied from interface: UnderFileSystem
        Renames a directory from src to dst in under file system.
        Parameters:
        src - the source directory path
        dst - the destination directory path
        Returns:
        true if succeed, false otherwise
        Throws:
        java.io.IOException

      • renameRenamableDirectory

        public boolean renameRenamableDirectory​(java.lang.String src,
                                        java.lang.String dst)
                                 throws java.io.IOException
        Description copied from interface: UnderFileSystem
        Renames a directory from src to dst in under file system. Similar to UnderFileSystem.renameDirectory(String, String) but deals with the write-src-then-rename and delete-dst-then-rename eventual consistency issue.
        Parameters:
        src - the source directory path
        dst - the destination directory path
        Returns:
        true if succeed, false otherwise
        Throws:
        java.io.IOException

      • renameFile

        public boolean renameFile​(java.lang.String src,
                          java.lang.String dst)
                   throws java.io.IOException
        Description copied from interface: UnderFileSystem
        Renames a file from src to dst in under file system.
        Parameters:
        src - the source file path
        dst - the destination file path
        Returns:
        true if succeed, false otherwise
        Throws:
        java.io.IOException

      • renameRenamableFile

        public boolean renameRenamableFile​(java.lang.String src,
                                   java.lang.String dst)
                            throws java.io.IOException
        Description copied from interface: UnderFileSystem
        Renames a file from src to dst in under file system. Similar to UnderFileSystem.renameFile(String, String) but deals with the write-src-then-rename and delete-dst-then-rename eventual consistency issue.
        Parameters:
        src - the source file path
        dst - the destination file path
        Returns:
        true if succeed, false otherwise
        Throws:
        java.io.IOException

      • supportsFlush

        public boolean supportsFlush()
                      throws java.io.IOException
        Description copied from interface: UnderFileSystem
        Whether this type of UFS supports flush.
        Returns:
        true if this type of UFS supports flush, false otherwise
        Throws:
        java.io.IOException

      • createEmptyObject

        public abstract boolean createEmptyObject​(java.lang.String key)
        Creates a zero-byte object used to encode a directory.
        Parameters:
        key - the key to create
        Returns:
        true if the operation was successful

      • createObject

        protected abstract java.io.OutputStream createObject​(java.lang.String key)
                                              throws java.io.IOException
        Creates an OutputStream for object uploads.
        Parameters:
        key - ufs key including scheme and bucket
        Returns:
        new OutputStream
        Throws:
        java.io.IOException

      • convertToFolderName

        protected java.lang.String convertToFolderName​(java.lang.String key)
        Appends the directory suffix to the key.
        Parameters:
        key - the key to convert
        Returns:
        key as a directory path

      • copyObject

        protected abstract boolean copyObject​(java.lang.String src,
                                      java.lang.String dst)
                               throws java.io.IOException
        Copies an object to another key.
        Parameters:
        src - the source key to copy
        dst - the destination key to copy to
        Returns:
        true if the operation was successful, false otherwise
        Throws:
        java.io.IOException

      • deleteObject

        protected abstract boolean deleteObject​(java.lang.String key)
                                 throws java.io.IOException
        Internal function to delete a key.
        Parameters:
        key - the key to delete
        Returns:
        true if successful, false if an exception is thrown
        Throws:
        java.io.IOException

      • deleteObjects

        protected java.util.List<java.lang.String> deleteObjects​(java.util.List<java.lang.String> keys)
                                                  throws java.io.IOException
        Internal function to delete a list of keys.
        Parameters:
        keys - the list of keys to delete
        Returns:
        list of successfully deleted keys
        Throws:
        java.io.IOException

      • getListingChunkLengthMax

        protected int getListingChunkLengthMax()
        Maximum number of items in a single listing chunk supported by the under store.
        Returns:
        the maximum length for a single listing query

      • getListingChunkLength

        protected int getListingChunkLength​(AlluxioConfiguration conf)
        The number of items to query in a single listing chunk.
        Returns:
        length of each list request

      • getObjectStatus

        @Nullable
protected abstract ObjectUnderFileSystem.ObjectStatus getObjectStatus​(java.lang.String key)
                                                               throws java.io.IOException
        Get metadata information about object. Implementations should process the key as is, which may be a file or a directory key.
        Parameters:
        key - ufs key to get metadata for
        Returns:
        ObjectUnderFileSystem.ObjectStatus if key exists and successful, otherwise null
        Throws:
        java.io.IOException

      • getParentPath

        @Nullable
protected java.lang.String getParentPath​(java.lang.String path)
        Get parent path.
        Parameters:
        path - ufs path including scheme and bucket
        Returns:
        the parent path, or null if the parent does not exist

      • isRoot

        protected boolean isRoot​(java.lang.String path)
        Checks if the path is the root. This method supports full path (e.g. s3://bucket_name/dir) and stripped path (e.g. /dir).
        Parameters:
        path - ufs path including scheme and bucket
        Returns:
        true if the path is the root, false otherwise

      • getChildName

        protected java.lang.String getChildName​(java.lang.String child,
                                        java.lang.String parent)
                                 throws java.io.IOException
        Gets the child name based on the parent name.
        Parameters:
        child - the key of the child
        parent - the key of the parent
        Returns:
        the child key with the parent prefix removed
        Throws:
        java.io.IOException

      • getFolderSuffix

        protected abstract java.lang.String getFolderSuffix()
        Get suffix used to encode a directory.
        Returns:
        folder suffix

      • getObjectListingChunk

        @Nullable
protected ObjectUnderFileSystem.ObjectListingChunk getObjectListingChunk​(java.lang.String key,
                                                                         boolean recursive,
                                                                         java.lang.String startAfter,
                                                                         int batchSize)
                                                                  throws java.io.IOException
        Gets a (partial) object listing result for the given key.
        Parameters:
        key - pseudo-directory key excluding header and bucket
        recursive - whether to request immediate children only, or all descendants
        startAfter - indicates where the listing starts
        batchSize - the batch size of each chunk
        Returns:
        chunked object listing, or null if key is not found
        Throws:
        java.io.IOException

      • getObjectListingChunk

        @Nullable
protected abstract ObjectUnderFileSystem.ObjectListingChunk getObjectListingChunk​(java.lang.String key,
                                                                                  boolean recursive)
                                                                           throws java.io.IOException
        Gets a (partial) object listing result for the given key.
        Parameters:
        key - pseudo-directory key excluding header and bucket
        recursive - whether to request immediate children only, or all descendants
        Returns:
        chunked object listing, or null if key is not found
        Throws:
        java.io.IOException

      • getObjectListingChunkForPath

        protected ObjectUnderFileSystem.ObjectListingChunk getObjectListingChunkForPath​(java.lang.String path,
                                                                                boolean recursive)
                                                                         throws java.io.IOException
        Throws:
        java.io.IOException

      • getObjectListingChunkForPath

        @Nullable
protected ObjectUnderFileSystem.ObjectListingChunk getObjectListingChunkForPath​(java.lang.String path,
                                                                                boolean recursive,
                                                                                java.lang.String startAfter,
                                                                                int batchSize)
                                                                         throws java.io.IOException
        Gets a (partial) object listing for the given path.
        Parameters:
        path - of pseudo-directory
        recursive - whether to request immediate children only, or all descendants
        Returns:
        chunked object listing, or null if the path does not exist as a pseudo-directory
        Throws:
        java.io.IOException

      • getRootKey

        protected abstract java.lang.String getRootKey()
        Get full path of root in object store.
        Returns:
        full path including scheme and bucket

      • listInternal

        @Nullable
protected UfsStatus[] listInternal​(java.lang.String path,
                                   ListOptions options)
                            throws java.io.IOException
        Lists the files in the given path, the paths will be their logical names and not contain the folder suffix. Note that, the list results are unsorted.
        Parameters:
        path - the key to list
        options - for listing
        Returns:
        an array of the file and folder names in this directory
        Throws:
        java.io.IOException

      • mkdirsInternal

        protected boolean mkdirsInternal​(java.lang.String key)
        Creates a directory flagged file with the key and folder suffix.
        Parameters:
        key - the key to create a folder
        Returns:
        true if the operation was successful, false otherwise

      • openObject

        protected abstract java.io.InputStream openObject​(java.lang.String key,
                                                  OpenOptions options,
                                                  RetryPolicy retryPolicy)
                                           throws java.io.IOException
        Internal function to open an input stream to an object.
        Parameters:
        key - the key to open
        options - the open options
        retryPolicy - the retry policy of the opened stream to solve eventual consistency issue
        Returns:
        an InputStream to read from key
        Throws:
        java.io.IOException

      • parentExists

        protected boolean parentExists​(java.lang.String path)
                        throws java.io.IOException
        Treating the object store as a file system, checks if the parent directory exists.
        Parameters:
        path - the path to check
        Returns:
        true if the parent exists or if the path is root, false otherwise
        Throws:
        java.io.IOException

      • stripPrefixIfPresent

        public java.lang.String stripPrefixIfPresent​(java.lang.String path)
        Strips the bucket prefix or the preceding path separator from the path if it is present. For example, for input path ufs://my-bucket-name/my-path/file, the output would be my-path/file. If path is an absolute path like /my-path/file, the output would be my-path/file. This method will leave keys without a prefix unaltered, ie. my-path/file returns my-path/file.
        Parameters:
        path - the path to strip
        Returns:
        the path without the bucket prefix

      • retryOnException

        protected <T> T retryOnException​(ObjectUnderFileSystem.ObjectStoreOperation<T> op,
                                 java.util.function.Supplier<java.lang.String> description)
                          throws java.io.IOException
        Retries the given object store operation when it throws exceptions to resolve eventual consistency issue.
        Parameters:
        op - the object store operation to retry
        description - the description regarding the operation
        Returns:
        the operation result if operation succeed
        Throws:
        java.io.IOException