Package alluxio.underfs.hdfs

Class HdfsUnderFileSystem

  • All Implemented Interfaces:
    alluxio.underfs.AtomicFileOutputStreamCallback, alluxio.underfs.UnderFileSystem, java.io.Closeable, java.lang.AutoCloseable
    @ThreadSafe
public class HdfsUnderFileSystem
extends alluxio.underfs.ConsistentUnderFileSystem
implements alluxio.underfs.AtomicFileOutputStreamCallback
    HDFS UnderFileSystem implementation.

    • Nested Class Summary

      • Nested classes/interfaces inherited from interface alluxio.underfs.UnderFileSystem

        alluxio.underfs.UnderFileSystem.Factory, alluxio.underfs.UnderFileSystem.SpaceType

    • Field Summary

      • Fields inherited from class alluxio.underfs.BaseUnderFileSystem

        EMPTY_ACL, mUfsConf, mUri

    • Constructor Summary

      Constructors 
      Constructor Description
      HdfsUnderFileSystem​(alluxio.AlluxioURI ufsUri, alluxio.underfs.UnderFileSystemConfiguration conf, org.apache.hadoop.conf.Configuration hdfsConf)
      Constructs a new HDFS UnderFileSystem.

    • Method Summary

      All Methods Static Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      void cleanup()  
      void close()  
      void connectFromMaster​(java.lang.String host)  
      void connectFromWorker​(java.lang.String host)  
      java.io.OutputStream create​(java.lang.String path, alluxio.underfs.options.CreateOptions options)  
      static org.apache.hadoop.conf.Configuration createConfiguration​(alluxio.underfs.UnderFileSystemConfiguration conf)
      Prepares the Hadoop configuration necessary to successfully obtain a FileSystem instance that can access the provided path.
      java.io.OutputStream createDirect​(java.lang.String path, alluxio.underfs.options.CreateOptions options)  
      static HdfsUnderFileSystem createInstance​(alluxio.AlluxioURI ufsUri, alluxio.underfs.UnderFileSystemConfiguration conf)
      Factory method to constructs a new HDFS UnderFileSystem instance.
      boolean deleteDirectory​(java.lang.String path, alluxio.underfs.options.DeleteOptions options)  
      boolean deleteFile​(java.lang.String path)  
      boolean exists​(java.lang.String path)  
      alluxio.collections.Pair<alluxio.security.authorization.AccessControlList,​alluxio.security.authorization.DefaultAccessControlList> getAclPair​(java.lang.String path)  
      alluxio.SyncInfo getActiveSyncInfo()  
      long getBlockSizeByte​(java.lang.String path)  
      alluxio.underfs.UfsDirectoryStatus getDirectoryStatus​(java.lang.String path)  
      java.util.List<java.lang.String> getFileLocations​(java.lang.String path)  
      java.util.List<java.lang.String> getFileLocations​(java.lang.String path, alluxio.underfs.options.FileLocationOptions options)  
      alluxio.underfs.UfsFileStatus getFileStatus​(java.lang.String path, alluxio.underfs.options.GetStatusOptions options)  
      long getSpace​(java.lang.String path, alluxio.underfs.UnderFileSystem.SpaceType type)  
      alluxio.underfs.UfsStatus getStatus​(java.lang.String path, alluxio.underfs.options.GetStatusOptions options)  
      java.lang.String getUnderFSType()  
      boolean isDirectory​(java.lang.String path)  
      boolean isFile​(java.lang.String path)  
      boolean isSeekable()  
      alluxio.underfs.UfsStatus[] listStatus​(java.lang.String path)  
      boolean mkdirs​(java.lang.String path, alluxio.underfs.options.MkdirsOptions options)  
      java.io.InputStream open​(java.lang.String path, alluxio.underfs.options.OpenOptions options)  
      HdfsPositionedUnderFileInputStream openPositionRead​(java.lang.String path, long fileLength)  
      boolean renameDirectory​(java.lang.String src, java.lang.String dst)  
      boolean renameFile​(java.lang.String src, java.lang.String dst)  
      void setAclEntries​(java.lang.String path, java.util.List<alluxio.security.authorization.AclEntry> aclEntries)  
      void setMode​(java.lang.String path, short mode)  
      void setOwner​(java.lang.String path, java.lang.String user, java.lang.String group)  
      boolean startActiveSyncPolling​(long txId)  
      void startSync​(alluxio.AlluxioURI ufsUri)  
      boolean stopActiveSyncPolling()  
      void stopSync​(alluxio.AlluxioURI ufsUri)  
      boolean supportsActiveSync()  
      boolean supportsFlush()  

      • Methods inherited from class alluxio.underfs.ConsistentUnderFileSystem

        createNonexistingFile, createNonexistingFile, deleteExistingDirectory, deleteExistingDirectory, deleteExistingFile, getExistingDirectoryStatus, getExistingFileStatus, getExistingStatus, isExistingDirectory, openExistingFile, openExistingFile, renameRenamableDirectory, renameRenamableFile

      • Methods inherited from class alluxio.underfs.BaseUnderFileSystem

        create, deleteDirectory, getConfiguration, getFingerprint, getOperationMode, getParsedFingerprint, getParsedFingerprint, getPhysicalStores, isObjectStorage, listStatus, mkdirs, open, resolveUri, validatePath

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

      • Methods inherited from interface alluxio.underfs.UnderFileSystem

        create, createNonexistingFile, createNonexistingFile, deleteDirectory, deleteExistingDirectory, deleteExistingDirectory, deleteExistingFile, getConfiguration, getExistingDirectoryStatus, getExistingFileStatus, getExistingStatus, getFileStatus, getFingerprint, getOperationMode, getParsedFingerprint, getParsedFingerprint, getPhysicalStores, getStatus, isExistingDirectory, isObjectStorage, listStatus, listStatuses, mkdirs, open, openExistingFile, openExistingFile, renameRenamableDirectory, renameRenamableFile, resolveUri

    • Constructor Detail

      • HdfsUnderFileSystem

        public HdfsUnderFileSystem​(alluxio.AlluxioURI ufsUri,
                           alluxio.underfs.UnderFileSystemConfiguration conf,
                           org.apache.hadoop.conf.Configuration hdfsConf)
        Constructs a new HDFS UnderFileSystem.
        Parameters:
        ufsUri - the AlluxioURI for this UFS
        conf - the configuration for this UFS
        hdfsConf - the configuration for HDFS

    • Method Detail

      • createInstance

        public static HdfsUnderFileSystem createInstance​(alluxio.AlluxioURI ufsUri,
                                                 alluxio.underfs.UnderFileSystemConfiguration conf)
        Factory method to constructs a new HDFS UnderFileSystem instance.
        Parameters:
        ufsUri - the AlluxioURI for this UFS
        conf - the configuration for Hadoop
        Returns:
        a new HDFS UnderFileSystem instance

      • getUnderFSType

        public java.lang.String getUnderFSType()
        Specified by:
        getUnderFSType in interface alluxio.underfs.UnderFileSystem

      • createConfiguration

        public static org.apache.hadoop.conf.Configuration createConfiguration​(alluxio.underfs.UnderFileSystemConfiguration conf)
        Prepares the Hadoop configuration necessary to successfully obtain a FileSystem instance that can access the provided path.

        Derived implementations that work with specialised Hadoop FileSystem API compatible implementations can override this method to add implementation specific configuration necessary for obtaining a usable FileSystem instance.

        Parameters:
        conf - the configuration for this UFS
        Returns:
        the configuration for HDFS

      • cleanup

        public void cleanup()
        Specified by:
        cleanup in interface alluxio.underfs.UnderFileSystem

      • close

        public void close()
        Specified by:
        close in interface java.lang.AutoCloseable
        Specified by:
        close in interface java.io.Closeable

      • create

        public java.io.OutputStream create​(java.lang.String path,
                                   alluxio.underfs.options.CreateOptions options)
                            throws java.io.IOException
        Specified by:
        create in interface alluxio.underfs.UnderFileSystem
        Throws:
        java.io.IOException

      • createDirect

        public java.io.OutputStream createDirect​(java.lang.String path,
                                         alluxio.underfs.options.CreateOptions options)
                                  throws java.io.IOException
        Specified by:
        createDirect in interface alluxio.underfs.AtomicFileOutputStreamCallback
        Throws:
        java.io.IOException

      • deleteDirectory

        public boolean deleteDirectory​(java.lang.String path,
                               alluxio.underfs.options.DeleteOptions options)
                        throws java.io.IOException
        Specified by:
        deleteDirectory in interface alluxio.underfs.UnderFileSystem
        Throws:
        java.io.IOException

      • deleteFile

        public boolean deleteFile​(java.lang.String path)
                   throws java.io.IOException
        Specified by:
        deleteFile in interface alluxio.underfs.UnderFileSystem
        Throws:
        java.io.IOException

      • exists

        public boolean exists​(java.lang.String path)
               throws java.io.IOException
        Specified by:
        exists in interface alluxio.underfs.UnderFileSystem
        Overrides:
        exists in class alluxio.underfs.BaseUnderFileSystem
        Throws:
        java.io.IOException

      • getAclPair

        public alluxio.collections.Pair<alluxio.security.authorization.AccessControlList,​alluxio.security.authorization.DefaultAccessControlList> getAclPair​(java.lang.String path)
                                                                                                                                                    throws java.io.IOException
        Specified by:
        getAclPair in interface alluxio.underfs.UnderFileSystem
        Overrides:
        getAclPair in class alluxio.underfs.BaseUnderFileSystem
        Throws:
        java.io.IOException

      • setAclEntries

        public void setAclEntries​(java.lang.String path,
                          java.util.List<alluxio.security.authorization.AclEntry> aclEntries)
                   throws java.io.IOException
        Specified by:
        setAclEntries in interface alluxio.underfs.UnderFileSystem
        Overrides:
        setAclEntries in class alluxio.underfs.BaseUnderFileSystem
        Throws:
        java.io.IOException

      • getBlockSizeByte

        public long getBlockSizeByte​(java.lang.String path)
                      throws java.io.IOException
        Specified by:
        getBlockSizeByte in interface alluxio.underfs.UnderFileSystem
        Throws:
        java.io.IOException

      • getDirectoryStatus

        public alluxio.underfs.UfsDirectoryStatus getDirectoryStatus​(java.lang.String path)
                                                      throws java.io.IOException
        Specified by:
        getDirectoryStatus in interface alluxio.underfs.UnderFileSystem
        Throws:
        java.io.IOException

      • getFileLocations

        public java.util.List<java.lang.String> getFileLocations​(java.lang.String path)
                                                  throws java.io.IOException
        Specified by:
        getFileLocations in interface alluxio.underfs.UnderFileSystem
        Throws:
        java.io.IOException

      • getFileLocations

        @Nullable
public java.util.List<java.lang.String> getFileLocations​(java.lang.String path,
                                                         alluxio.underfs.options.FileLocationOptions options)
                                                  throws java.io.IOException
        Specified by:
        getFileLocations in interface alluxio.underfs.UnderFileSystem
        Throws:
        java.io.IOException

      • getFileStatus

        public alluxio.underfs.UfsFileStatus getFileStatus​(java.lang.String path,
                                                   alluxio.underfs.options.GetStatusOptions options)
                                            throws java.io.IOException
        Specified by:
        getFileStatus in interface alluxio.underfs.UnderFileSystem
        Throws:
        java.io.IOException

      • getSpace

        public long getSpace​(java.lang.String path,
                     alluxio.underfs.UnderFileSystem.SpaceType type)
              throws java.io.IOException
        Specified by:
        getSpace in interface alluxio.underfs.UnderFileSystem
        Throws:
        java.io.IOException

      • getStatus

        public alluxio.underfs.UfsStatus getStatus​(java.lang.String path,
                                           alluxio.underfs.options.GetStatusOptions options)
                                    throws java.io.IOException
        Specified by:
        getStatus in interface alluxio.underfs.UnderFileSystem
        Throws:
        java.io.IOException

      • isDirectory

        public boolean isDirectory​(java.lang.String path)
                    throws java.io.IOException
        Specified by:
        isDirectory in interface alluxio.underfs.UnderFileSystem
        Throws:
        java.io.IOException

      • isFile

        public boolean isFile​(java.lang.String path)
               throws java.io.IOException
        Specified by:
        isFile in interface alluxio.underfs.UnderFileSystem
        Throws:
        java.io.IOException

      • listStatus

        @Nullable
public alluxio.underfs.UfsStatus[] listStatus​(java.lang.String path)
                                       throws java.io.IOException
        Specified by:
        listStatus in interface alluxio.underfs.UnderFileSystem
        Throws:
        java.io.IOException

      • connectFromMaster

        public void connectFromMaster​(java.lang.String host)
                       throws java.io.IOException
        Specified by:
        connectFromMaster in interface alluxio.underfs.UnderFileSystem
        Throws:
        java.io.IOException

      • connectFromWorker

        public void connectFromWorker​(java.lang.String host)
                       throws java.io.IOException
        Specified by:
        connectFromWorker in interface alluxio.underfs.UnderFileSystem
        Throws:
        java.io.IOException

      • mkdirs

        public boolean mkdirs​(java.lang.String path,
                      alluxio.underfs.options.MkdirsOptions options)
               throws java.io.IOException
        Specified by:
        mkdirs in interface alluxio.underfs.UnderFileSystem
        Throws:
        java.io.IOException

      • open

        public java.io.InputStream open​(java.lang.String path,
                                alluxio.underfs.options.OpenOptions options)
                         throws java.io.IOException
        Specified by:
        open in interface alluxio.underfs.UnderFileSystem
        Throws:
        java.io.IOException

      • openPositionRead

        public HdfsPositionedUnderFileInputStream openPositionRead​(java.lang.String path,
                                                           long fileLength)
        Specified by:
        openPositionRead in interface alluxio.underfs.UnderFileSystem

      • renameDirectory

        public boolean renameDirectory​(java.lang.String src,
                               java.lang.String dst)
                        throws java.io.IOException
        Specified by:
        renameDirectory in interface alluxio.underfs.UnderFileSystem
        Throws:
        java.io.IOException

      • renameFile

        public boolean renameFile​(java.lang.String src,
                          java.lang.String dst)
                   throws java.io.IOException
        Specified by:
        renameFile in interface alluxio.underfs.UnderFileSystem
        Throws:
        java.io.IOException

      • setOwner

        public void setOwner​(java.lang.String path,
                     java.lang.String user,
                     java.lang.String group)
              throws java.io.IOException
        Specified by:
        setOwner in interface alluxio.underfs.UnderFileSystem
        Throws:
        java.io.IOException

      • setMode

        public void setMode​(java.lang.String path,
                    short mode)
             throws java.io.IOException
        Specified by:
        setMode in interface alluxio.underfs.UnderFileSystem
        Throws:
        java.io.IOException

      • supportsFlush

        public boolean supportsFlush()
        Specified by:
        supportsFlush in interface alluxio.underfs.UnderFileSystem

      • supportsActiveSync

        public boolean supportsActiveSync()
        Specified by:
        supportsActiveSync in interface alluxio.underfs.UnderFileSystem
        Overrides:
        supportsActiveSync in class alluxio.underfs.BaseUnderFileSystem

      • getActiveSyncInfo

        public alluxio.SyncInfo getActiveSyncInfo()
        Specified by:
        getActiveSyncInfo in interface alluxio.underfs.UnderFileSystem
        Overrides:
        getActiveSyncInfo in class alluxio.underfs.BaseUnderFileSystem

      • startActiveSyncPolling

        public boolean startActiveSyncPolling​(long txId)
                               throws java.io.IOException
        Specified by:
        startActiveSyncPolling in interface alluxio.underfs.UnderFileSystem
        Overrides:
        startActiveSyncPolling in class alluxio.underfs.BaseUnderFileSystem
        Throws:
        java.io.IOException

      • stopActiveSyncPolling

        public boolean stopActiveSyncPolling()
        Specified by:
        stopActiveSyncPolling in interface alluxio.underfs.UnderFileSystem
        Overrides:
        stopActiveSyncPolling in class alluxio.underfs.BaseUnderFileSystem

      • startSync

        public void startSync​(alluxio.AlluxioURI ufsUri)
        Specified by:
        startSync in interface alluxio.underfs.UnderFileSystem
        Overrides:
        startSync in class alluxio.underfs.BaseUnderFileSystem

      • stopSync

        public void stopSync​(alluxio.AlluxioURI ufsUri)
        Specified by:
        stopSync in interface alluxio.underfs.UnderFileSystem
        Overrides:
        stopSync in class alluxio.underfs.BaseUnderFileSystem

      • isSeekable

        public boolean isSeekable()
        Specified by:
        isSeekable in interface alluxio.underfs.UnderFileSystem
        Overrides:
        isSeekable in class alluxio.underfs.BaseUnderFileSystem