org.apache.hadoop.hbase.io.hfile

Class HFileReaderImpl

java.lang.Object

org.apache.hadoop.hbase.io.hfile.HFileReaderImpl

All Implemented Interfaces:

Closeable, AutoCloseable, org.apache.hadoop.conf.Configurable, HFile.CachingBlockReader, HFile.Reader

@InterfaceAudience.Private
public class HFileReaderImpl
extends Object
implements HFile.Reader, org.apache.hadoop.conf.Configurable

Implementation that can handle all hfile versions of HFile.Reader.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	HFileReaderImpl.BlockIndexNotLoadedException
protected static class	HFileReaderImpl.EncodedScanner Scanner that operates on encoded data blocks.
protected static class	HFileReaderImpl.HFileScannerImpl
static class	HFileReaderImpl.NotSeekedException An exception thrown when an operation requiring a scanner to be seeked is invoked on a scanner that is not seeked.

Field Summary

Fields

Modifier and Type	Field and Description
protected boolean	decodeMemstoreTS
static int	KEY_VALUE_LEN_SIZE The size of a (key length, value length) tuple that prefixes each entry in a data block.
static int	MINOR_VERSION_NO_CHECKSUM In HFile minor version that does not support checksums
static int	MINOR_VERSION_WITH_CHECKSUM Minor versions in HFile starting with this number have hbase checksums
static int	PBUF_TRAILER_MINOR_VERSION HFile minor version that introduced pbuf filetrailer

Constructor Summary

Constructors

Constructor and Description
HFileReaderImpl(org.apache.hadoop.fs.Path path, FixedFileTrailer trailer, FSDataInputStreamWrapper fsdis, long fileSize, CacheConfig cacheConf, HFileSystem hfs, boolean primaryReplicaReader, org.apache.hadoop.conf.Configuration conf) Opens a HFile.
HFileReaderImpl(org.apache.hadoop.fs.Path path, FixedFileTrailer trailer, FSDataInputStreamWrapper fsdis, long fileSize, CacheConfig cacheConf, HFileSystem hfs, org.apache.hadoop.conf.Configuration conf) Deprecated.

Method Summary

Modifier and Type	Method and Description
void	close()
void	close(boolean evictOnClose) Close method with optional evictOnClose
protected HFileContext	createHFileContext(FSDataInputStreamWrapper fsdis, long fileSize, HFileSystem hfs, org.apache.hadoop.fs.Path path, FixedFileTrailer trailer)
CellComparator	getComparator()
Compression.Algorithm	getCompressionAlgorithm()
org.apache.hadoop.conf.Configuration	getConf()
DataBlockEncoding	getDataBlockEncoding()
org.apache.hadoop.hbase.io.hfile.HFileBlockIndex.BlockIndexReader	getDataBlockIndexReader()
DataInput	getDeleteBloomFilterMetadata() Retrieves delete family Bloom filter metadata as appropriate for each HFile version.
DataBlockEncoding	getEffectiveEncodingInCache(boolean isCompaction)
long	getEntries()
HFileContext	getFileContext() Return the file context of the HFile this reader belongs to
Optional<Cell>	getFirstKey()
Optional<byte[]>	getFirstRowKey() TODO left from HFile version 1: move this to StoreFile after Ryan's patch goes in to eliminate KeyValue here.
DataInput	getGeneralBloomFilterMetadata() Returns a buffer with the Bloom filter metadata.
Optional<Cell>	getLastKey()
Optional<byte[]>	getLastRowKey() TODO left from HFile version 1: move this to StoreFile after Ryan's patch goes in to eliminate KeyValue here.
int	getMajorVersion()
HFileBlock	getMetaBlock(String metaBlockName, boolean cacheBlock)
String	getName() Returns this reader's "name".
org.apache.hadoop.fs.Path	getPath()
HFileScanner	getScanner(boolean cacheBlocks, boolean pread) Create a Scanner on this file.
HFileScanner	getScanner(boolean cacheBlocks, boolean pread, boolean isCompaction) Create a Scanner on this file.
FixedFileTrailer	getTrailer()
org.apache.hadoop.hbase.io.hfile.HFileBlock.FSReader	getUncachedBlockReader() For testing
boolean	hasMVCCInfo()
long	indexSize()
boolean	isDecodeMemStoreTS()
boolean	isFileInfoLoaded()
boolean	isPrimaryReplicaReader()
long	length()
HFile.FileInfo	loadFileInfo()
Optional<Cell>	midKey()
boolean	prefetchComplete() Returns false if block prefetching was requested for this file and has not completed, true otherwise
HFileBlock	readBlock(long dataBlockOffset, long onDiskBlockSize, boolean cacheBlock, boolean pread, boolean isCompaction, boolean updateCacheMetrics, BlockType expectedBlockType, DataBlockEncoding expectedDataBlockEncoding) Read in a file block.
void	returnBlock(HFileBlock block) Return the given block back to the cache, if it was obtained from cache.
void	setConf(org.apache.hadoop.conf.Configuration conf)
boolean	shouldIncludeMemStoreTS()
String	toString()
void	unbufferStream() To close the stream's socket.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

MINOR_VERSION_WITH_CHECKSUM

public static final int MINOR_VERSION_WITH_CHECKSUM

Minor versions in HFile starting with this number have hbase checksums

See Also:

Constant Field Values

MINOR_VERSION_NO_CHECKSUM

public static final int MINOR_VERSION_NO_CHECKSUM

In HFile minor version that does not support checksums

See Also:

Constant Field Values

PBUF_TRAILER_MINOR_VERSION

public static final int PBUF_TRAILER_MINOR_VERSION

HFile minor version that introduced pbuf filetrailer

See Also:

Constant Field Values

KEY_VALUE_LEN_SIZE

public static final int KEY_VALUE_LEN_SIZE

The size of a (key length, value length) tuple that prefixes each entry in a data block.

See Also:

Constant Field Values

decodeMemstoreTS

protected boolean decodeMemstoreTS

Constructor Detail

HFileReaderImpl

@Deprecated
public HFileReaderImpl(org.apache.hadoop.fs.Path path,
                                   FixedFileTrailer trailer,
                                   FSDataInputStreamWrapper fsdis,
                                   long fileSize,
                                   CacheConfig cacheConf,
                                   HFileSystem hfs,
                                   org.apache.hadoop.conf.Configuration conf)
                            throws IOException

Deprecated.

Throws:

IOException

HFileReaderImpl

public HFileReaderImpl(org.apache.hadoop.fs.Path path,
                       FixedFileTrailer trailer,
                       FSDataInputStreamWrapper fsdis,
                       long fileSize,
                       CacheConfig cacheConf,
                       HFileSystem hfs,
                       boolean primaryReplicaReader,
                       org.apache.hadoop.conf.Configuration conf)
                throws IOException

Opens a HFile. You must load the index before you can use it by calling loadFileInfo().

Parameters:

path - Path to HFile.

trailer - File trailer.

fsdis - input stream.

fileSize - Length of the stream.

cacheConf - Cache configuration.

hfs - The file system.

conf - Configuration

Throws:

IOException

Method Detail

toString

public String toString()

Overrides:

toString in class Object

length

public long length()

Specified by:

length in interface HFile.Reader

returnBlock

public void returnBlock(HFileBlock block)

Description copied from interface: HFile.CachingBlockReader

Return the given block back to the cache, if it was obtained from cache.

Specified by:

returnBlock in interface HFile.CachingBlockReader

Parameters:

block - Block to be returned.

getFirstKey

public Optional<Cell> getFirstKey()

Specified by:

getFirstKey in interface HFile.Reader

Returns:

the first key in the file. May be null if file has no entries. Note that this is not the first row key, but rather the byte form of the first KeyValue.

getFirstRowKey

public Optional<byte[]> getFirstRowKey()

TODO left from HFile version 1: move this to StoreFile after Ryan's patch goes in to eliminate KeyValue here.

Specified by:

getFirstRowKey in interface HFile.Reader

Returns:

the first row key, or null if the file is empty.

getLastRowKey

public Optional<byte[]> getLastRowKey()

TODO left from HFile version 1: move this to StoreFile after Ryan's patch goes in to eliminate KeyValue here.

Specified by:

getLastRowKey in interface HFile.Reader

Returns:

the last row key, or null if the file is empty.

getEntries

public long getEntries()

Specified by:

getEntries in interface HFile.Reader

Returns:

number of KV entries in this HFile

getComparator

public CellComparator getComparator()

Specified by:

getComparator in interface HFile.Reader

Returns:

comparator

getCompressionAlgorithm

public Compression.Algorithm getCompressionAlgorithm()

Specified by:

getCompressionAlgorithm in interface HFile.Reader

Returns:

compression algorithm

indexSize

public long indexSize()

Specified by:

indexSize in interface HFile.Reader

Returns:

the total heap size of data and meta block indexes in bytes. Does not take into account non-root blocks of a multilevel data index.

getName

public String getName()

Description copied from interface: HFile.Reader

Returns this reader's "name". Usually the last component of the path. Needs to be constant as the file is being moved to support caching on write.

Specified by:

getName in interface HFile.Reader

getDataBlockIndexReader

public org.apache.hadoop.hbase.io.hfile.HFileBlockIndex.BlockIndexReader getDataBlockIndexReader()

Specified by:

getDataBlockIndexReader in interface HFile.Reader

getTrailer

public FixedFileTrailer getTrailer()

Specified by:

getTrailer in interface HFile.Reader

isPrimaryReplicaReader

public boolean isPrimaryReplicaReader()

Specified by:

isPrimaryReplicaReader in interface HFile.Reader

loadFileInfo

public HFile.FileInfo loadFileInfo()
                            throws IOException

Specified by:

loadFileInfo in interface HFile.Reader

Throws:

IOException

getPath

public org.apache.hadoop.fs.Path getPath()

Specified by:

getPath in interface HFile.Reader

getDataBlockEncoding

public DataBlockEncoding getDataBlockEncoding()

Specified by:

getDataBlockEncoding in interface HFile.Reader

getConf

public org.apache.hadoop.conf.Configuration getConf()

Specified by:

getConf in interface org.apache.hadoop.conf.Configurable

setConf

public void setConf(org.apache.hadoop.conf.Configuration conf)

Specified by:

setConf in interface org.apache.hadoop.conf.Configurable

isDecodeMemStoreTS

public boolean isDecodeMemStoreTS()

Specified by:

isDecodeMemStoreTS in interface HFile.Reader

shouldIncludeMemStoreTS

public boolean shouldIncludeMemStoreTS()

Specified by:

shouldIncludeMemStoreTS in interface HFile.Reader

getMetaBlock

public HFileBlock getMetaBlock(String metaBlockName,
                               boolean cacheBlock)
                        throws IOException

Specified by:

getMetaBlock in interface HFile.Reader

Parameters:

metaBlockName -

cacheBlock - Add block to cache, if found

Returns:

block wrapped in a ByteBuffer, with header skipped

Throws:

IOException

readBlock

public HFileBlock readBlock(long dataBlockOffset,
                            long onDiskBlockSize,
                            boolean cacheBlock,
                            boolean pread,
                            boolean isCompaction,
                            boolean updateCacheMetrics,
                            BlockType expectedBlockType,
                            DataBlockEncoding expectedDataBlockEncoding)
                     throws IOException

Description copied from interface: HFile.CachingBlockReader

Read in a file block.

Specified by:

readBlock in interface HFile.CachingBlockReader

Parameters:

dataBlockOffset - offset to read.

onDiskBlockSize - size of the block

isCompaction - is this block being read as part of a compaction

expectedBlockType - the block type we are expecting to read with this read operation, or null to read whatever block type is available and avoid checking (that might reduce caching efficiency of encoded data blocks)

expectedDataBlockEncoding - the data block encoding the caller is expecting data blocks to be in, or null to not perform this check and return the block irrespective of the encoding. This check only applies to data blocks and can be set to null when the caller is expecting to read a non-data block and has set expectedBlockType accordingly.

Returns:

Block wrapped in a ByteBuffer.

Throws:

IOException

hasMVCCInfo

public boolean hasMVCCInfo()

Specified by:

hasMVCCInfo in interface HFile.Reader

getLastKey

public Optional<Cell> getLastKey()

Specified by:

getLastKey in interface HFile.Reader

Returns:

Last key as cell in the file. May be null if file has no entries. Note that this is not the last row key, but it is the Cell representation of the last key

midKey

public Optional<Cell> midKey()
                      throws IOException

Specified by:

midKey in interface HFile.Reader

Returns:

Midkey for this file. We work with block boundaries only so returned midkey is an approximation only.

Throws:

IOException

close

public void close()
           throws IOException

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Throws:

IOException

close

public void close(boolean evictOnClose)
           throws IOException

Description copied from interface: HFile.Reader

Close method with optional evictOnClose

Specified by:

close in interface HFile.Reader

Throws:

IOException

getEffectiveEncodingInCache

public DataBlockEncoding getEffectiveEncodingInCache(boolean isCompaction)

Specified by:

getEffectiveEncodingInCache in interface HFile.Reader

getUncachedBlockReader

public org.apache.hadoop.hbase.io.hfile.HFileBlock.FSReader getUncachedBlockReader()

For testing

Specified by:

getUncachedBlockReader in interface HFile.Reader

getGeneralBloomFilterMetadata

public DataInput getGeneralBloomFilterMetadata()
                                        throws IOException

Returns a buffer with the Bloom filter metadata. The caller takes ownership of the buffer.

Specified by:

getGeneralBloomFilterMetadata in interface HFile.Reader

Throws:

IOException

getDeleteBloomFilterMetadata

public DataInput getDeleteBloomFilterMetadata()
                                       throws IOException

Description copied from interface: HFile.Reader

Retrieves delete family Bloom filter metadata as appropriate for each HFile version. Knows nothing about how that metadata is structured.

Specified by:

getDeleteBloomFilterMetadata in interface HFile.Reader

Throws:

IOException

isFileInfoLoaded

public boolean isFileInfoLoaded()

getFileContext

public HFileContext getFileContext()

Description copied from interface: HFile.Reader

Return the file context of the HFile this reader belongs to

Specified by:

getFileContext in interface HFile.Reader

prefetchComplete

public boolean prefetchComplete()

Returns false if block prefetching was requested for this file and has not completed, true otherwise

Specified by:

prefetchComplete in interface HFile.Reader

createHFileContext

protected HFileContext createHFileContext(FSDataInputStreamWrapper fsdis,
                                          long fileSize,
                                          HFileSystem hfs,
                                          org.apache.hadoop.fs.Path path,
                                          FixedFileTrailer trailer)
                                   throws IOException

Throws:

IOException

getScanner

public HFileScanner getScanner(boolean cacheBlocks,
                               boolean pread)

Create a Scanner on this file. No seeks or reads are done on creation. Call HFileScanner.seekTo(Cell) to position an start the read. There is nothing to clean up in a Scanner. Letting go of your references to the scanner is sufficient. NOTE: Do not use this overload of getScanner for compactions. See getScanner(boolean, boolean, boolean)

Specified by:

getScanner in interface HFile.Reader

Parameters:

cacheBlocks - True if we should cache blocks read in by this scanner.

pread - Use positional read rather than seek+read if true (pread is better for random reads, seek+read is better scanning).

Returns:

Scanner on this file.

getScanner

public HFileScanner getScanner(boolean cacheBlocks,
                               boolean pread,
                               boolean isCompaction)

Create a Scanner on this file. No seeks or reads are done on creation. Call HFileScanner.seekTo(Cell) to position an start the read. There is nothing to clean up in a Scanner. Letting go of your references to the scanner is sufficient.

Specified by:

getScanner in interface HFile.Reader

Parameters:

cacheBlocks - True if we should cache blocks read in by this scanner.

pread - Use positional read rather than seek+read if true (pread is better for random reads, seek+read is better scanning).

isCompaction - is scanner being used for a compaction?

Returns:

Scanner on this file.

getMajorVersion

public int getMajorVersion()

unbufferStream

public void unbufferStream()

Description copied from interface: HFile.Reader

To close the stream's socket. Note: This can be concurrently called from multiple threads and implementation should take care of thread safety.

Specified by:

unbufferStream in interface HFile.Reader