org.apache.hadoop.hdfs.server.datanode

Class LocalReplica

java.lang.Object

org.apache.hadoop.hdfs.protocol.Block

org.apache.hadoop.hdfs.server.datanode.ReplicaInfo

org.apache.hadoop.hdfs.server.datanode.LocalReplica

All Implemented Interfaces:

Comparable<org.apache.hadoop.hdfs.protocol.Block>, org.apache.hadoop.hdfs.server.datanode.Replica, org.apache.hadoop.io.Writable, org.apache.hadoop.util.LightWeightGSet.LinkedElement

Direct Known Subclasses:

FinalizedReplica, LocalReplicaInPipeline, ReplicaUnderRecovery, ReplicaWaitingToBeRecovered

public abstract class LocalReplica
extends org.apache.hadoop.hdfs.server.datanode.ReplicaInfo

This class is used for all replicas which are on local storage media and hence, are backed by files.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	LocalReplica.ReplicaDirInfo

Field Summary

Fields inherited from class org.apache.hadoop.hdfs.protocol.Block

BLOCK_FILE_PREFIX, blockFilePattern, METADATA_EXTENSION, metaFilePattern, metaOrBlockFilePattern

Method Summary

Modifier and Type	Method and Description
boolean	blockDataExists()
boolean	breakHardLinksIfNeeded() This function "breaks hardlinks" to the current replica file.
void	bumpReplicaGS(long newGS) Bump a replica's generation stamp to a new one.
int	compareWith(FsVolumeSpi.ScanInfo info)
void	copyBlockdata(URI destination)
void	copyMetadata(URI destination)
boolean	deleteBlockData() Used to deletes the replica's block data.
boolean	deleteMetadata() Used to deletes the replica's metadata.
void	fsyncDirectory() Sync the parent directory changes to durable device.
long	getBlockDataLength()
File	getBlockFile() Get the full path of this replica's data file.
URI	getBlockURI() Get the URI for where the data of this replica is stored.
InputStream	getDataInputStream(long seekOffset) Returns an InputStream to the replica's data.
OutputStream	getDataOutputStream(boolean append) Returns an OutputStream to the replica's data.
protected File	getDir() Return the parent directory path where this replica is located.
LengthInputStream	getMetadataInputStream(long offset) Returns an InputStream to the replica's metadata.
long	getMetadataLength()
OutputStream	getMetadataOutputStream(boolean append) Returns an OutputStream to the replica's metadata.
URI	getMetadataURI() Get the URI for where the metadata of this replica is stored.
File	getMetaFile() Get the full path of this replica's meta file.
boolean	getPinning(org.apache.hadoop.fs.LocalFileSystem localFS) Check whether the block was pinned.
boolean	getPinning(org.apache.hadoop.fs.LocalFileSystem localFS, org.apache.hadoop.fs.Path path) Get pin status of a file by checking the sticky bit.
boolean	metadataExists()
static LocalReplica.ReplicaDirInfo	parseBaseDir(File dir)
boolean	renameData(URI destURI) Rename the data URI to that referenced by destURI.
boolean	renameMeta(URI destURI) Rename the metadata URI to that referenced by destURI.
void	setPinning(org.apache.hadoop.fs.LocalFileSystem localFS) Set a block to be pinned on this datanode so that it cannot be moved by Balancer/Mover.
void	setPinning(org.apache.hadoop.fs.LocalFileSystem localFS, org.apache.hadoop.fs.Path path) Set sticky bit on path to pin file.
static void	truncateBlock(FsVolumeSpi volume, File blockFile, File metaFile, long oldlen, long newlen, org.apache.hadoop.hdfs.server.datanode.FileIoProvider fileIoProvider)
void	truncateBlock(long newLength)
void	updateWithReplica(org.apache.hadoop.hdfs.server.datanode.StorageLocation replicaLocation) Update this replica with the StorageLocation found.

Methods inherited from class org.apache.hadoop.hdfs.server.datanode.ReplicaInfo

createInfo, getBytesReserved, getFileIoProvider, getNext, getOriginalBytesReserved, getOriginalReplica, getRecoveryID, getStorageUuid, getVolume, isOnTransientStorage, setNext, setRecoveryID, toString

Methods inherited from class org.apache.hadoop.hdfs.protocol.Block

appendStringTo, compareTo, equals, filename2id, getBlockId, getBlockId, getBlockName, getGenerationStamp, getGenerationStamp, getNumBytes, hashCode, isBlockFilename, isMetaFilename, matchingIdAndGenStamp, metaToBlockFile, readFields, readId, set, setBlockId, setGenerationStamp, setNumBytes, toString, write, writeId

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.hadoop.hdfs.server.datanode.Replica

getBlockId, getBytesOnDisk, getGenerationStamp, getNumBytes, getState, getVisibleLength

Method Detail

getBlockFile

public File getBlockFile()

Get the full path of this replica's data file.

Returns:

the full path of this replica's data file

getMetaFile

public File getMetaFile()

Get the full path of this replica's meta file.

Returns:

the full path of this replica's meta file

getDir

protected File getDir()

Return the parent directory path where this replica is located.

Returns:

the parent directory path where this replica is located

parseBaseDir

public static LocalReplica.ReplicaDirInfo parseBaseDir(File dir)

breakHardLinksIfNeeded

public boolean breakHardLinksIfNeeded()
                               throws IOException

This function "breaks hardlinks" to the current replica file. When doing a DataNode upgrade, we create a bunch of hardlinks to each block file. This cleverly ensures that both the old and the new storage directories can contain the same block file, without using additional space for the data. However, when we want to append to the replica file, we need to "break" the hardlink to ensure that the old snapshot continues to contain the old data length. If we failed to do that, we could roll back to the previous/ directory during a downgrade, and find that the block contents were longer than they were at the time of upgrade.

Specified by:

breakHardLinksIfNeeded in class org.apache.hadoop.hdfs.server.datanode.ReplicaInfo

Returns:

true only if data was copied.

Throws:

IOException

getBlockURI

public URI getBlockURI()

Description copied from class: org.apache.hadoop.hdfs.server.datanode.ReplicaInfo

Get the URI for where the data of this replica is stored.

Specified by:

getBlockURI in class org.apache.hadoop.hdfs.server.datanode.ReplicaInfo

Returns:

URI for the location of replica data.

getDataInputStream

public InputStream getDataInputStream(long seekOffset)
                               throws IOException

Description copied from class: org.apache.hadoop.hdfs.server.datanode.ReplicaInfo

Returns an InputStream to the replica's data.

Specified by:

getDataInputStream in class org.apache.hadoop.hdfs.server.datanode.ReplicaInfo

Parameters:

seekOffset - the offset at which the read is started from.

Returns:

the InputStream to read the replica data.

Throws:

IOException - if an error occurs in opening a stream to the data.

getDataOutputStream

public OutputStream getDataOutputStream(boolean append)
                                 throws IOException

Description copied from class: org.apache.hadoop.hdfs.server.datanode.ReplicaInfo

Returns an OutputStream to the replica's data.

Specified by:

getDataOutputStream in class org.apache.hadoop.hdfs.server.datanode.ReplicaInfo

Parameters:

append - indicates if the block should be opened for append.

Returns:

the OutputStream to write to the replica.

Throws:

IOException - if an error occurs in creating an OutputStream.

blockDataExists

public boolean blockDataExists()

Specified by:

blockDataExists in class org.apache.hadoop.hdfs.server.datanode.ReplicaInfo

Returns:

true if the replica's data exists.

deleteBlockData

public boolean deleteBlockData()

Description copied from class: org.apache.hadoop.hdfs.server.datanode.ReplicaInfo

Used to deletes the replica's block data.

Specified by:

deleteBlockData in class org.apache.hadoop.hdfs.server.datanode.ReplicaInfo

Returns:

true if the replica's data is successfully deleted.

getBlockDataLength

public long getBlockDataLength()

Specified by:

getBlockDataLength in class org.apache.hadoop.hdfs.server.datanode.ReplicaInfo

Returns:

the length of the block on storage.

getMetadataURI

public URI getMetadataURI()

Description copied from class: org.apache.hadoop.hdfs.server.datanode.ReplicaInfo

Get the URI for where the metadata of this replica is stored.

Specified by:

getMetadataURI in class org.apache.hadoop.hdfs.server.datanode.ReplicaInfo

Returns:

URI for the location of replica metadata.

getMetadataInputStream

public LengthInputStream getMetadataInputStream(long offset)
                                         throws IOException

Description copied from class: org.apache.hadoop.hdfs.server.datanode.ReplicaInfo

Returns an InputStream to the replica's metadata.

Specified by:

getMetadataInputStream in class org.apache.hadoop.hdfs.server.datanode.ReplicaInfo

Parameters:

offset - the offset at which the read is started from.

Returns:

the LengthInputStream to read the replica metadata.

Throws:

IOException

getMetadataOutputStream

public OutputStream getMetadataOutputStream(boolean append)
                                     throws IOException

Description copied from class: org.apache.hadoop.hdfs.server.datanode.ReplicaInfo

Returns an OutputStream to the replica's metadata.

Specified by:

getMetadataOutputStream in class org.apache.hadoop.hdfs.server.datanode.ReplicaInfo

Parameters:

append - indicates if the block metadata should be opened for append.

Returns:

the OutputStream to write to the replica's metadata.

Throws:

IOException - if an error occurs in creating an OutputStream.

metadataExists

public boolean metadataExists()

Specified by:

metadataExists in class org.apache.hadoop.hdfs.server.datanode.ReplicaInfo

Returns:

true if the replica's metadata exists.

deleteMetadata

public boolean deleteMetadata()

Description copied from class: org.apache.hadoop.hdfs.server.datanode.ReplicaInfo

Used to deletes the replica's metadata.

Specified by:

deleteMetadata in class org.apache.hadoop.hdfs.server.datanode.ReplicaInfo

Returns:

true if the replica's metadata is successfully deleted.

getMetadataLength

public long getMetadataLength()

Specified by:

getMetadataLength in class org.apache.hadoop.hdfs.server.datanode.ReplicaInfo

Returns:

the length of the metadata on storage.

renameMeta

public boolean renameMeta(URI destURI)
                   throws IOException

Description copied from class: org.apache.hadoop.hdfs.server.datanode.ReplicaInfo

Rename the metadata URI to that referenced by destURI.

Specified by:

renameMeta in class org.apache.hadoop.hdfs.server.datanode.ReplicaInfo

Parameters:

destURI - the target URI.

Returns:

true if the rename is successful.

Throws:

IOException - if an exception occurs in the rename.

renameData

public boolean renameData(URI destURI)
                   throws IOException

Description copied from class: org.apache.hadoop.hdfs.server.datanode.ReplicaInfo

Rename the data URI to that referenced by destURI.

Specified by:

renameData in class org.apache.hadoop.hdfs.server.datanode.ReplicaInfo

Parameters:

destURI - the target URI.

Returns:

true if the rename is successful.

Throws:

IOException - if an exception occurs in the rename.

updateWithReplica

public void updateWithReplica(org.apache.hadoop.hdfs.server.datanode.StorageLocation replicaLocation)

Description copied from class: org.apache.hadoop.hdfs.server.datanode.ReplicaInfo

Update this replica with the StorageLocation found.

Specified by:

updateWithReplica in class org.apache.hadoop.hdfs.server.datanode.ReplicaInfo

Parameters:

replicaLocation - the StorageLocation found for this replica.

getPinning

public boolean getPinning(org.apache.hadoop.fs.LocalFileSystem localFS)
                   throws IOException

Description copied from class: org.apache.hadoop.hdfs.server.datanode.ReplicaInfo

Check whether the block was pinned.

Specified by:

getPinning in class org.apache.hadoop.hdfs.server.datanode.ReplicaInfo

Parameters:

localFS - the local filesystem to use.

Returns:

true if the block is pinned.

Throws:

IOException

setPinning

public void setPinning(org.apache.hadoop.fs.LocalFileSystem localFS)
                throws IOException

Description copied from class: org.apache.hadoop.hdfs.server.datanode.ReplicaInfo

Set a block to be pinned on this datanode so that it cannot be moved by Balancer/Mover.

Specified by:

setPinning in class org.apache.hadoop.hdfs.server.datanode.ReplicaInfo

Parameters:

localFS - the local filesystem to use.

Throws:

IOException - if there is an exception in the pinning.

bumpReplicaGS

public void bumpReplicaGS(long newGS)
                   throws IOException

Description copied from class: org.apache.hadoop.hdfs.server.datanode.ReplicaInfo

Bump a replica's generation stamp to a new one. Its on-disk meta file name is renamed to be the new one too.

Specified by:

bumpReplicaGS in class org.apache.hadoop.hdfs.server.datanode.ReplicaInfo

Parameters:

newGS - new generation stamp

Throws:

IOException - if the change fails

truncateBlock

public void truncateBlock(long newLength)
                   throws IOException

Specified by:

truncateBlock in class org.apache.hadoop.hdfs.server.datanode.ReplicaInfo

Throws:

IOException

compareWith

public int compareWith(FsVolumeSpi.ScanInfo info)

Specified by:

compareWith in class org.apache.hadoop.hdfs.server.datanode.ReplicaInfo

copyMetadata

public void copyMetadata(URI destination)
                  throws IOException

Specified by:

copyMetadata in class org.apache.hadoop.hdfs.server.datanode.ReplicaInfo

Throws:

IOException

copyBlockdata

public void copyBlockdata(URI destination)
                   throws IOException

Specified by:

copyBlockdata in class org.apache.hadoop.hdfs.server.datanode.ReplicaInfo

Throws:

IOException

getPinning

public boolean getPinning(org.apache.hadoop.fs.LocalFileSystem localFS,
                          org.apache.hadoop.fs.Path path)
                   throws IOException

Get pin status of a file by checking the sticky bit.

Parameters:

localFS - local file system

path - path to be checked

Returns:

true if the file is pinned with sticky bit

Throws:

IOException

setPinning

public void setPinning(org.apache.hadoop.fs.LocalFileSystem localFS,
                       org.apache.hadoop.fs.Path path)
                throws IOException

Set sticky bit on path to pin file.

Parameters:

localFS - local file system

path - path to be pinned with sticky bit

Throws:

IOException

truncateBlock

public static void truncateBlock(FsVolumeSpi volume,
                                 File blockFile,
                                 File metaFile,
                                 long oldlen,
                                 long newlen,
                                 org.apache.hadoop.hdfs.server.datanode.FileIoProvider fileIoProvider)
                          throws IOException

Throws:

IOException

fsyncDirectory

public void fsyncDirectory()
                    throws IOException

Sync the parent directory changes to durable device.

Throws:

IOException