Package alluxio.underfs.hdfs
Class HdfsUnderFileSystem
- java.lang.Object
-
- alluxio.underfs.BaseUnderFileSystem
-
- alluxio.underfs.ConsistentUnderFileSystem
-
- alluxio.underfs.hdfs.HdfsUnderFileSystem
-
- All Implemented Interfaces:
alluxio.underfs.AtomicFileOutputStreamCallback
,alluxio.underfs.UnderFileSystem
,java.io.Closeable
,java.lang.AutoCloseable
@ThreadSafe public class HdfsUnderFileSystem extends alluxio.underfs.ConsistentUnderFileSystem implements alluxio.underfs.AtomicFileOutputStreamCallback
HDFSUnderFileSystem
implementation.
-
-
Constructor Summary
Constructors Constructor Description HdfsUnderFileSystem(alluxio.AlluxioURI ufsUri, alluxio.underfs.UnderFileSystemConfiguration conf, org.apache.hadoop.conf.Configuration hdfsConf)
Constructs a new HDFSUnderFileSystem
.
-
Method Summary
All Methods Static Methods Instance Methods Concrete Methods Modifier and Type Method Description void
cleanup()
void
close()
void
connectFromMaster(java.lang.String host)
void
connectFromWorker(java.lang.String host)
java.io.OutputStream
create(java.lang.String path, alluxio.underfs.options.CreateOptions options)
static org.apache.hadoop.conf.Configuration
createConfiguration(alluxio.underfs.UnderFileSystemConfiguration conf)
Prepares the Hadoop configuration necessary to successfully obtain aFileSystem
instance that can access the provided path.java.io.OutputStream
createDirect(java.lang.String path, alluxio.underfs.options.CreateOptions options)
static HdfsUnderFileSystem
createInstance(alluxio.AlluxioURI ufsUri, alluxio.underfs.UnderFileSystemConfiguration conf)
Factory method to constructs a new HDFSUnderFileSystem
instance.boolean
deleteDirectory(java.lang.String path, alluxio.underfs.options.DeleteOptions options)
boolean
deleteFile(java.lang.String path)
boolean
exists(java.lang.String path)
alluxio.collections.Pair<alluxio.security.authorization.AccessControlList,alluxio.security.authorization.DefaultAccessControlList>
getAclPair(java.lang.String path)
alluxio.SyncInfo
getActiveSyncInfo()
long
getBlockSizeByte(java.lang.String path)
alluxio.underfs.UfsDirectoryStatus
getDirectoryStatus(java.lang.String path)
java.util.List<java.lang.String>
getFileLocations(java.lang.String path)
java.util.List<java.lang.String>
getFileLocations(java.lang.String path, alluxio.underfs.options.FileLocationOptions options)
alluxio.underfs.UfsFileStatus
getFileStatus(java.lang.String path)
long
getSpace(java.lang.String path, alluxio.underfs.UnderFileSystem.SpaceType type)
alluxio.underfs.UfsStatus
getStatus(java.lang.String path)
java.lang.String
getUnderFSType()
boolean
isDirectory(java.lang.String path)
boolean
isFile(java.lang.String path)
boolean
isSeekable()
alluxio.underfs.UfsStatus[]
listStatus(java.lang.String path)
boolean
mkdirs(java.lang.String path, alluxio.underfs.options.MkdirsOptions options)
java.io.InputStream
open(java.lang.String path, alluxio.underfs.options.OpenOptions options)
HdfsPositionedUnderFileInputStream
openPositionRead(java.lang.String path, long fileLength)
boolean
renameDirectory(java.lang.String src, java.lang.String dst)
boolean
renameFile(java.lang.String src, java.lang.String dst)
void
setAclEntries(java.lang.String path, java.util.List<alluxio.security.authorization.AclEntry> aclEntries)
void
setMode(java.lang.String path, short mode)
void
setOwner(java.lang.String path, java.lang.String user, java.lang.String group)
boolean
startActiveSyncPolling(long txId)
void
startSync(alluxio.AlluxioURI ufsUri)
boolean
stopActiveSyncPolling()
void
stopSync(alluxio.AlluxioURI ufsUri)
boolean
supportsActiveSync()
boolean
supportsFlush()
-
Methods inherited from class alluxio.underfs.ConsistentUnderFileSystem
createNonexistingFile, createNonexistingFile, deleteExistingDirectory, deleteExistingDirectory, deleteExistingFile, getExistingDirectoryStatus, getExistingFileStatus, getExistingStatus, isExistingDirectory, openExistingFile, openExistingFile, renameRenamableDirectory, renameRenamableFile
-
Methods inherited from class alluxio.underfs.BaseUnderFileSystem
create, deleteDirectory, getConfiguration, getFingerprint, getOperationMode, getParsedFingerprint, getPhysicalStores, isObjectStorage, listStatus, mkdirs, open, resolveUri, validatePath
-
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
-
Methods inherited from interface alluxio.underfs.UnderFileSystem
create, createNonexistingFile, createNonexistingFile, deleteDirectory, deleteExistingDirectory, deleteExistingDirectory, deleteExistingFile, getConfiguration, getExistingDirectoryStatus, getExistingFileStatus, getExistingStatus, getFingerprint, getOperationMode, getParsedFingerprint, getPhysicalStores, isExistingDirectory, isObjectStorage, listStatus, listStatuses, mkdirs, open, openExistingFile, openExistingFile, renameRenamableDirectory, renameRenamableFile, resolveUri
-
-
-
-
Constructor Detail
-
HdfsUnderFileSystem
public HdfsUnderFileSystem(alluxio.AlluxioURI ufsUri, alluxio.underfs.UnderFileSystemConfiguration conf, org.apache.hadoop.conf.Configuration hdfsConf)
Constructs a new HDFSUnderFileSystem
.- Parameters:
ufsUri
- theAlluxioURI
for this UFSconf
- the configuration for this UFShdfsConf
- the configuration for HDFS
-
-
Method Detail
-
createInstance
public static HdfsUnderFileSystem createInstance(alluxio.AlluxioURI ufsUri, alluxio.underfs.UnderFileSystemConfiguration conf)
Factory method to constructs a new HDFSUnderFileSystem
instance.- Parameters:
ufsUri
- theAlluxioURI
for this UFSconf
- the configuration for Hadoop- Returns:
- a new HDFS
UnderFileSystem
instance
-
getUnderFSType
public java.lang.String getUnderFSType()
- Specified by:
getUnderFSType
in interfacealluxio.underfs.UnderFileSystem
-
createConfiguration
public static org.apache.hadoop.conf.Configuration createConfiguration(alluxio.underfs.UnderFileSystemConfiguration conf)
Prepares the Hadoop configuration necessary to successfully obtain aFileSystem
instance that can access the provided path.Derived implementations that work with specialised Hadoop FileSystem API compatible implementations can override this method to add implementation specific configuration necessary for obtaining a usable FileSystem instance.
- Parameters:
conf
- the configuration for this UFS- Returns:
- the configuration for HDFS
-
cleanup
public void cleanup()
- Specified by:
cleanup
in interfacealluxio.underfs.UnderFileSystem
-
close
public void close()
- Specified by:
close
in interfacejava.lang.AutoCloseable
- Specified by:
close
in interfacejava.io.Closeable
-
create
public java.io.OutputStream create(java.lang.String path, alluxio.underfs.options.CreateOptions options) throws java.io.IOException
- Specified by:
create
in interfacealluxio.underfs.UnderFileSystem
- Throws:
java.io.IOException
-
createDirect
public java.io.OutputStream createDirect(java.lang.String path, alluxio.underfs.options.CreateOptions options) throws java.io.IOException
- Specified by:
createDirect
in interfacealluxio.underfs.AtomicFileOutputStreamCallback
- Throws:
java.io.IOException
-
deleteDirectory
public boolean deleteDirectory(java.lang.String path, alluxio.underfs.options.DeleteOptions options) throws java.io.IOException
- Specified by:
deleteDirectory
in interfacealluxio.underfs.UnderFileSystem
- Throws:
java.io.IOException
-
deleteFile
public boolean deleteFile(java.lang.String path) throws java.io.IOException
- Specified by:
deleteFile
in interfacealluxio.underfs.UnderFileSystem
- Throws:
java.io.IOException
-
exists
public boolean exists(java.lang.String path) throws java.io.IOException
- Specified by:
exists
in interfacealluxio.underfs.UnderFileSystem
- Overrides:
exists
in classalluxio.underfs.BaseUnderFileSystem
- Throws:
java.io.IOException
-
getAclPair
public alluxio.collections.Pair<alluxio.security.authorization.AccessControlList,alluxio.security.authorization.DefaultAccessControlList> getAclPair(java.lang.String path) throws java.io.IOException
- Specified by:
getAclPair
in interfacealluxio.underfs.UnderFileSystem
- Overrides:
getAclPair
in classalluxio.underfs.BaseUnderFileSystem
- Throws:
java.io.IOException
-
setAclEntries
public void setAclEntries(java.lang.String path, java.util.List<alluxio.security.authorization.AclEntry> aclEntries) throws java.io.IOException
- Specified by:
setAclEntries
in interfacealluxio.underfs.UnderFileSystem
- Overrides:
setAclEntries
in classalluxio.underfs.BaseUnderFileSystem
- Throws:
java.io.IOException
-
getBlockSizeByte
public long getBlockSizeByte(java.lang.String path) throws java.io.IOException
- Specified by:
getBlockSizeByte
in interfacealluxio.underfs.UnderFileSystem
- Throws:
java.io.IOException
-
getDirectoryStatus
public alluxio.underfs.UfsDirectoryStatus getDirectoryStatus(java.lang.String path) throws java.io.IOException
- Specified by:
getDirectoryStatus
in interfacealluxio.underfs.UnderFileSystem
- Throws:
java.io.IOException
-
getFileLocations
public java.util.List<java.lang.String> getFileLocations(java.lang.String path) throws java.io.IOException
- Specified by:
getFileLocations
in interfacealluxio.underfs.UnderFileSystem
- Throws:
java.io.IOException
-
getFileLocations
@Nullable public java.util.List<java.lang.String> getFileLocations(java.lang.String path, alluxio.underfs.options.FileLocationOptions options) throws java.io.IOException
- Specified by:
getFileLocations
in interfacealluxio.underfs.UnderFileSystem
- Throws:
java.io.IOException
-
getFileStatus
public alluxio.underfs.UfsFileStatus getFileStatus(java.lang.String path) throws java.io.IOException
- Specified by:
getFileStatus
in interfacealluxio.underfs.UnderFileSystem
- Throws:
java.io.IOException
-
getSpace
public long getSpace(java.lang.String path, alluxio.underfs.UnderFileSystem.SpaceType type) throws java.io.IOException
- Specified by:
getSpace
in interfacealluxio.underfs.UnderFileSystem
- Throws:
java.io.IOException
-
getStatus
public alluxio.underfs.UfsStatus getStatus(java.lang.String path) throws java.io.IOException
- Specified by:
getStatus
in interfacealluxio.underfs.UnderFileSystem
- Throws:
java.io.IOException
-
isDirectory
public boolean isDirectory(java.lang.String path) throws java.io.IOException
- Specified by:
isDirectory
in interfacealluxio.underfs.UnderFileSystem
- Throws:
java.io.IOException
-
isFile
public boolean isFile(java.lang.String path) throws java.io.IOException
- Specified by:
isFile
in interfacealluxio.underfs.UnderFileSystem
- Throws:
java.io.IOException
-
listStatus
@Nullable public alluxio.underfs.UfsStatus[] listStatus(java.lang.String path) throws java.io.IOException
- Specified by:
listStatus
in interfacealluxio.underfs.UnderFileSystem
- Throws:
java.io.IOException
-
connectFromMaster
public void connectFromMaster(java.lang.String host) throws java.io.IOException
- Specified by:
connectFromMaster
in interfacealluxio.underfs.UnderFileSystem
- Throws:
java.io.IOException
-
connectFromWorker
public void connectFromWorker(java.lang.String host) throws java.io.IOException
- Specified by:
connectFromWorker
in interfacealluxio.underfs.UnderFileSystem
- Throws:
java.io.IOException
-
mkdirs
public boolean mkdirs(java.lang.String path, alluxio.underfs.options.MkdirsOptions options) throws java.io.IOException
- Specified by:
mkdirs
in interfacealluxio.underfs.UnderFileSystem
- Throws:
java.io.IOException
-
open
public java.io.InputStream open(java.lang.String path, alluxio.underfs.options.OpenOptions options) throws java.io.IOException
- Specified by:
open
in interfacealluxio.underfs.UnderFileSystem
- Throws:
java.io.IOException
-
openPositionRead
public HdfsPositionedUnderFileInputStream openPositionRead(java.lang.String path, long fileLength)
- Specified by:
openPositionRead
in interfacealluxio.underfs.UnderFileSystem
-
renameDirectory
public boolean renameDirectory(java.lang.String src, java.lang.String dst) throws java.io.IOException
- Specified by:
renameDirectory
in interfacealluxio.underfs.UnderFileSystem
- Throws:
java.io.IOException
-
renameFile
public boolean renameFile(java.lang.String src, java.lang.String dst) throws java.io.IOException
- Specified by:
renameFile
in interfacealluxio.underfs.UnderFileSystem
- Throws:
java.io.IOException
-
setOwner
public void setOwner(java.lang.String path, java.lang.String user, java.lang.String group) throws java.io.IOException
- Specified by:
setOwner
in interfacealluxio.underfs.UnderFileSystem
- Throws:
java.io.IOException
-
setMode
public void setMode(java.lang.String path, short mode) throws java.io.IOException
- Specified by:
setMode
in interfacealluxio.underfs.UnderFileSystem
- Throws:
java.io.IOException
-
supportsFlush
public boolean supportsFlush()
- Specified by:
supportsFlush
in interfacealluxio.underfs.UnderFileSystem
-
supportsActiveSync
public boolean supportsActiveSync()
- Specified by:
supportsActiveSync
in interfacealluxio.underfs.UnderFileSystem
- Overrides:
supportsActiveSync
in classalluxio.underfs.BaseUnderFileSystem
-
getActiveSyncInfo
public alluxio.SyncInfo getActiveSyncInfo()
- Specified by:
getActiveSyncInfo
in interfacealluxio.underfs.UnderFileSystem
- Overrides:
getActiveSyncInfo
in classalluxio.underfs.BaseUnderFileSystem
-
startActiveSyncPolling
public boolean startActiveSyncPolling(long txId) throws java.io.IOException
- Specified by:
startActiveSyncPolling
in interfacealluxio.underfs.UnderFileSystem
- Overrides:
startActiveSyncPolling
in classalluxio.underfs.BaseUnderFileSystem
- Throws:
java.io.IOException
-
stopActiveSyncPolling
public boolean stopActiveSyncPolling()
- Specified by:
stopActiveSyncPolling
in interfacealluxio.underfs.UnderFileSystem
- Overrides:
stopActiveSyncPolling
in classalluxio.underfs.BaseUnderFileSystem
-
startSync
public void startSync(alluxio.AlluxioURI ufsUri)
- Specified by:
startSync
in interfacealluxio.underfs.UnderFileSystem
- Overrides:
startSync
in classalluxio.underfs.BaseUnderFileSystem
-
stopSync
public void stopSync(alluxio.AlluxioURI ufsUri)
- Specified by:
stopSync
in interfacealluxio.underfs.UnderFileSystem
- Overrides:
stopSync
in classalluxio.underfs.BaseUnderFileSystem
-
isSeekable
public boolean isSeekable()
- Specified by:
isSeekable
in interfacealluxio.underfs.UnderFileSystem
- Overrides:
isSeekable
in classalluxio.underfs.BaseUnderFileSystem
-
-