HFileWriterImpl (Apache HBase - Server 2.2.2 API)

java.lang.Object
- org.apache.hadoop.hbase.io.hfile.HFileWriterImpl

All Implemented Interfaces:

Closeable, AutoCloseable, HFile.Writer, CellSink, ShipperListener
```
@InterfaceAudience.Private
public class HFileWriterImpl
extends Object
implements HFile.Writer
```
Common functionality needed by all versions of HFile writers.

Field Summary

Fields
Modifier and Type	Field and Description
`protected HFileDataBlockEncoder`	`blockEncoder` The data block encoding which will be used.
`protected org.apache.hadoop.hbase.io.hfile.HFileBlock.Writer`	`blockWriter` block writer
`protected CacheConfig`	`cacheConf` Cache configuration for caching data on write.
`protected boolean`	`closeOutputStream` True if we opened the `outputStream` (and so will close it).
`protected CellComparator`	`comparator` Key comparator.
`protected long`	`entryCount` Total # of key/value entries, i.e.
`protected HFile.FileInfo`	`fileInfo` A "file info" block: a key-value map of file-wide metadata.
`protected Cell`	`firstCellInBlock` First cell in a block.
`protected HFileContext`	`hFileContext`
`static int`	`KEY_VALUE_VER_WITH_MEMSTORE` Version for KeyValue which includes memstore timestamp
`static byte[]`	`KEY_VALUE_VERSION` KeyValue version in FileInfo
`protected Cell`	`lastCell` The Cell previously appended.
`protected long`	`lastDataBlockOffset` The offset of the last data block or 0 if the file is empty.
`protected long`	`maxMemstoreTS`
`protected List<org.apache.hadoop.io.Writable>`	`metaData` `Writable`s representing meta block data.
`protected List<byte[]>`	`metaNames` Meta block names.
`protected String`	`name` Name for this object used when logging or in toString.
`protected org.apache.hadoop.fs.FSDataOutputStream`	`outputStream` FileSystem stream to write into.
`protected org.apache.hadoop.fs.Path`	`path` May be null if we were passed a stream.
`protected long`	`totalKeyLength` Used for calculating the average key length.
`protected long`	`totalUncompressedBytes` Total uncompressed bytes, maybe calculate a compression ratio later.
`protected long`	`totalValueLength` Used for calculating the average value length.
`static String`	`UNIFIED_ENCODED_BLOCKSIZE_RATIO` if this feature is enabled, preCalculate encoded data size before real encoding happens

Fields inherited from interface org.apache.hadoop.hbase.io.hfile.HFile.Writer
MAX_MEMSTORE_TS_KEY

Constructor Summary

Constructors
Constructor and Description
`HFileWriterImpl(org.apache.hadoop.conf.Configuration conf, CacheConfig cacheConf, org.apache.hadoop.fs.Path path, org.apache.hadoop.fs.FSDataOutputStream outputStream, CellComparator comparator, HFileContext fileContext)`

Method Summary

All Methods Static Methods Instance Methods Concrete Methods
Modifier and Type	Method and Description
`void`	`addDeleteFamilyBloomFilter(BloomFilterWriter bfw)` Store delete family Bloom filter in the file, which is only supported in HFile V2.
`void`	`addGeneralBloomFilter(BloomFilterWriter bfw)` Store general Bloom filter in the file.
`void`	`addInlineBlockWriter(InlineBlockWriter ibw)` Adds an inline block writer such as a multi-level block index writer or a compound Bloom filter writer.
`void`	`append(Cell cell)` Add key/value to file.
`void`	`appendFileInfo(byte[] k, byte[] v)` Add to the file info.
`void`	`appendMetaBlock(String metaBlockName, org.apache.hadoop.io.Writable content)` Add a meta block to the end of the file.
`void`	`beforeShipped()` The action that needs to be performed before `Shipper.shipped()` is performed
`protected void`	`checkBlockBoundary()` At a block boundary, write all the inline blocks and opens new block.
`protected boolean`	`checkKey(Cell cell)` Checks that the given Cell's key does not violate the key order.
`protected void`	`checkValue(byte[] value, int offset, int length)` Checks the given value for validity.
`void`	`close()`
`static Compression.Algorithm`	`compressionByName(String algoName)`
`protected static org.apache.hadoop.fs.FSDataOutputStream`	`createOutputStream(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path, InetSocketAddress[] favoredNodes)` A helper method to create HFile output streams in constructors
`protected void`	`finishClose(FixedFileTrailer trailer)`
`protected void`	`finishFileInfo()`
`protected void`	`finishInit(org.apache.hadoop.conf.Configuration conf)` Additional initialization steps
`HFileContext`	`getFileContext()` Return the file context for the HFile this writer belongs to
`Cell`	`getLastCell()`
`protected int`	`getMajorVersion()`
`static Cell`	`getMidpoint(CellComparator comparator, Cell left, Cell right)` Try to return a Cell that falls between `left` and `right` but that is shorter; i.e.
`protected int`	`getMinorVersion()`
`org.apache.hadoop.fs.Path`	`getPath()`
`protected void`	`newBlock()` Ready a new block for writing.
`String`	`toString()`
`protected void`	`writeFileInfo(FixedFileTrailer trailer, DataOutputStream out)` Sets the file info offset in the trailer, finishes up populating fields in the file info, and writes the file info into the given data output.

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

- Field Detail
  - UNIFIED_ENCODED_BLOCKSIZE_RATIO
```
public static final String UNIFIED_ENCODED_BLOCKSIZE_RATIO
```
    if this feature is enabled, preCalculate encoded data size before real encoding happens
    
    See Also:
    
    Constant Field Values
  - lastCell
```
protected Cell lastCell
```
    The Cell previously appended. Becomes the last cell in the file.
  - outputStream
```
protected org.apache.hadoop.fs.FSDataOutputStream outputStream
```
    FileSystem stream to write into.
  - closeOutputStream
```
protected final boolean closeOutputStream
```
    True if we opened the outputStream (and so will close it).
  - fileInfo
```
protected HFile.FileInfo fileInfo
```
    A "file info" block: a key-value map of file-wide metadata.
  - entryCount
```
protected long entryCount
```
    Total # of key/value entries, i.e. how many times add() was called.
  - totalKeyLength
```
protected long totalKeyLength
```
    Used for calculating the average key length.
  - totalValueLength
```
protected long totalValueLength
```
    Used for calculating the average value length.
  - totalUncompressedBytes
```
protected long totalUncompressedBytes
```
    Total uncompressed bytes, maybe calculate a compression ratio later.
  - comparator
```
protected final CellComparator comparator
```
    Key comparator. Used to ensure we write in order.
  - metaNames
```
protected List<byte[]> metaNames
```
    Meta block names.
  - metaData
```
protected List<org.apache.hadoop.io.Writable> metaData
```
    Writables representing meta block data.
  - firstCellInBlock
```
protected Cell firstCellInBlock
```
    First cell in a block. This reference should be short-lived since we write hfiles in a burst.
  - path
```
protected final org.apache.hadoop.fs.Path path
```
    May be null if we were passed a stream.
  - cacheConf
```
protected final CacheConfig cacheConf
```
    Cache configuration for caching data on write.
  - name
```
protected final String name
```
    Name for this object used when logging or in toString. Is either the result of a toString on stream or else name of passed file Path.
  - blockEncoder
```
protected final HFileDataBlockEncoder blockEncoder
```
    The data block encoding which will be used. NoOpDataBlockEncoder.INSTANCE if there is no encoding.
  - hFileContext
```
protected final HFileContext hFileContext
```
  - KEY_VALUE_VERSION
```
public static final byte[] KEY_VALUE_VERSION
```
    KeyValue version in FileInfo
  - KEY_VALUE_VER_WITH_MEMSTORE
```
public static final int KEY_VALUE_VER_WITH_MEMSTORE
```
    Version for KeyValue which includes memstore timestamp
    
    See Also:
    
    Constant Field Values
  - blockWriter
```
protected org.apache.hadoop.hbase.io.hfile.HFileBlock.Writer blockWriter
```
    block writer
  - lastDataBlockOffset
```
protected long lastDataBlockOffset
```
    The offset of the last data block or 0 if the file is empty.
  - maxMemstoreTS
```
protected long maxMemstoreTS
```
- Constructor Detail
  - HFileWriterImpl
```
public HFileWriterImpl(org.apache.hadoop.conf.Configuration conf,
                       CacheConfig cacheConf,
                       org.apache.hadoop.fs.Path path,
                       org.apache.hadoop.fs.FSDataOutputStream outputStream,
                       CellComparator comparator,
                       HFileContext fileContext)
```
- Method Detail
  - appendFileInfo
```
public void appendFileInfo(byte[] k,
                           byte[] v)
                    throws IOException
```
    Add to the file info. All added key/value pairs can be obtained using HFile.Reader.loadFileInfo().
    
    Specified by:
    
    appendFileInfo in interface HFile.Writer
    
    Parameters:
    
    k - Key
    
    v - Value
    
    Throws:
    
    IOException - in case the key or the value are invalid
  - writeFileInfo
```
protected final void writeFileInfo(FixedFileTrailer trailer,
                                   DataOutputStream out)
                            throws IOException
```
    Sets the file info offset in the trailer, finishes up populating fields in the file info, and writes the file info into the given data output. The reason the data output is not always outputStream is that we store file info as a block in version 2.
    
    Parameters:
    
    trailer - fixed file trailer
    
    out - the data output to write the file info to
    
    Throws:
    
    IOException
  - checkKey
```
protected boolean checkKey(Cell cell)
                    throws IOException
```
    Checks that the given Cell's key does not violate the key order.
    
    Parameters:
    
    cell - Cell whose key to check.
    
    Returns:
    
    true if the key is duplicate
    
    Throws:
    
    IOException - if the key or the key order is wrong
  - checkValue
```
protected void checkValue(byte[] value,
                          int offset,
                          int length)
                   throws IOException
```
    Checks the given value for validity.
    
    Throws:
    
    IOException
  - getPath
```
public org.apache.hadoop.fs.Path getPath()
```
    Specified by:
    
    getPath in interface HFile.Writer
    
    Returns:
    
    Path or null if we were passed a stream rather than a Path.
  - toString
```
public String toString()
```
    Overrides:
    
    toString in class Object
  - compressionByName
```
public static Compression.Algorithm compressionByName(String algoName)
```
  - createOutputStream
```
protected static org.apache.hadoop.fs.FSDataOutputStream createOutputStream(org.apache.hadoop.conf.Configuration conf,
                                                                            org.apache.hadoop.fs.FileSystem fs,
                                                                            org.apache.hadoop.fs.Path path,
                                                                            InetSocketAddress[] favoredNodes)
                                                                     throws IOException
```
    A helper method to create HFile output streams in constructors
    
    Throws:
    
    IOException
  - finishInit
```
protected void finishInit(org.apache.hadoop.conf.Configuration conf)
```
    Additional initialization steps
  - checkBlockBoundary
```
protected void checkBlockBoundary()
                           throws IOException
```
    At a block boundary, write all the inline blocks and opens new block.
    
    Throws:
    
    IOException
  - getMidpoint
```
public static Cell getMidpoint(CellComparator comparator,
                               Cell left,
                               Cell right)
```
    Try to return a Cell that falls between left and right but that is shorter; i.e. takes up less space. This trick is used building HFile block index. Its an optimization. It does not always work. In this case we'll just return the right cell.
    
    Parameters:
    
    comparator - Comparator to use.
    
    left -
    
    right -
    
    Returns:
    
    A cell that sorts between left and right.
  - newBlock
```
protected void newBlock()
                 throws IOException
```
    Ready a new block for writing.
    
    Throws:
    
    IOException
  - appendMetaBlock
```
public void appendMetaBlock(String metaBlockName,
                            org.apache.hadoop.io.Writable content)
```
    Add a meta block to the end of the file. Call before close(). Metadata blocks are expensive. Fill one with a bunch of serialized data rather than do a metadata block per metadata instance. If metadata is small, consider adding to file info using appendFileInfo(byte[], byte[])
    
    Specified by:
    
    appendMetaBlock in interface HFile.Writer
    
    Parameters:
    
    metaBlockName - name of the block
    
    content - will call readFields to get data later (DO NOT REUSE)
  - close
```
public void close()
           throws IOException
```
    Specified by:
    
    close in interface Closeable
    
    Specified by:
    
    close in interface AutoCloseable
    
    Throws:
    
    IOException
  - addInlineBlockWriter
```
public void addInlineBlockWriter(InlineBlockWriter ibw)
```
    Description copied from interface: HFile.Writer
    
    Adds an inline block writer such as a multi-level block index writer or a compound Bloom filter writer.
    
    Specified by:
    
    addInlineBlockWriter in interface HFile.Writer
  - addGeneralBloomFilter
```
public void addGeneralBloomFilter(BloomFilterWriter bfw)
```
    Description copied from interface: HFile.Writer
    
    Store general Bloom filter in the file. This does not deal with Bloom filter internals but is necessary, since Bloom filters are stored differently in HFile version 1 and version 2.
    
    Specified by:
    
    addGeneralBloomFilter in interface HFile.Writer
  - addDeleteFamilyBloomFilter
```
public void addDeleteFamilyBloomFilter(BloomFilterWriter bfw)
```
    Description copied from interface: HFile.Writer
    
    Store delete family Bloom filter in the file, which is only supported in HFile V2.
    
    Specified by:
    
    addDeleteFamilyBloomFilter in interface HFile.Writer
  - getFileContext
```
public HFileContext getFileContext()
```
    Description copied from interface: HFile.Writer
    
    Return the file context for the HFile this writer belongs to
    
    Specified by:
    
    getFileContext in interface HFile.Writer
  - append
```
public void append(Cell cell)
            throws IOException
```
    Add key/value to file. Keys must be added in an order that agrees with the Comparator passed on construction.
    
    Specified by:
    
    append in interface CellSink
    
    Parameters:
    
    cell - Cell to add. Cannot be empty nor null.
    
    Throws:
    
    IOException
  - beforeShipped
```
public void beforeShipped()
                   throws IOException
```
    Description copied from interface: ShipperListener
    
    The action that needs to be performed before Shipper.shipped() is performed
    
    Specified by:
    
    beforeShipped in interface ShipperListener
    
    Throws:
    
    IOException
  - getLastCell
```
public Cell getLastCell()
```
  - finishFileInfo
```
protected void finishFileInfo()
                       throws IOException
```
    Throws:
    
    IOException
  - getMajorVersion
```
protected int getMajorVersion()
```
  - getMinorVersion
```
protected int getMinorVersion()
```
  - finishClose
```
protected void finishClose(FixedFileTrailer trailer)
                    throws IOException
```
    Throws:
    
    IOException

Class HFileWriterImpl

Field Summary

Fields inherited from interface org.apache.hadoop.hbase.io.hfile.HFile.Writer

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Field Detail

UNIFIED_ENCODED_BLOCKSIZE_RATIO

lastCell

outputStream

closeOutputStream

fileInfo

entryCount

totalKeyLength

totalValueLength

totalUncompressedBytes

comparator

metaNames

metaData

firstCellInBlock

path

cacheConf

name

blockEncoder

hFileContext

KEY_VALUE_VERSION

KEY_VALUE_VER_WITH_MEMSTORE

blockWriter

lastDataBlockOffset

maxMemstoreTS

Constructor Detail

HFileWriterImpl

Method Detail

appendFileInfo

writeFileInfo

checkKey

checkValue

getPath

toString

compressionByName

createOutputStream

finishInit

checkBlockBoundary

getMidpoint

newBlock

appendMetaBlock

close

addInlineBlockWriter

addGeneralBloomFilter

addDeleteFamilyBloomFilter

getFileContext

append

beforeShipped

getLastCell

finishFileInfo

getMajorVersion

getMinorVersion

finishClose