com.facebook.presto.operator.repartition

Class AbstractBlockEncodingBuffer

java.lang.Object

com.facebook.presto.operator.repartition.AbstractBlockEncodingBuffer

All Implemented Interfaces:

BlockEncodingBuffer

Direct Known Subclasses:

ArrayBlockEncodingBuffer, ByteArrayBlockEncodingBuffer, Int128ArrayBlockEncodingBuffer, IntArrayBlockEncodingBuffer, LongArrayBlockEncodingBuffer, MapBlockEncodingBuffer, RowBlockEncodingBuffer, ShortArrayBlockEncodingBuffer, VariableWidthBlockEncodingBuffer

public abstract class AbstractBlockEncodingBuffer
extends Object
implements BlockEncodingBuffer

Field Summary

Fields

Modifier and Type	Field and Description
protected int	batchSize
protected ArrayAllocator	bufferAllocator
protected int	bufferedPositionCount
protected Block	decodedBlock
protected boolean	flushed
static double	GRACE_FACTOR_FOR_MAX_BUFFER_CAPACITY
protected boolean	isNested
protected int	positionCount
protected boolean	positionsMapped
protected int	positionsOffset

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	AbstractBlockEncodingBuffer(ArrayAllocator bufferAllocator, boolean isNested)

Method Summary

Modifier and Type	Method and Description
protected abstract void	accumulateSerializedRowSizes(int[] positionOffsets, int positionCount, int[] serializedRowSizes) Called by accumulateSerializedRowSizes(int[] serializedRowSizes) for composite types to add row sizes for nested blocks.
protected void	appendNulls()
protected void	appendPositionRange(int offset, int length)
static BlockEncodingBuffer	createBlockEncodingBuffers(com.facebook.presto.operator.repartition.DecodedBlockNode decodedBlockNode, ArrayAllocator bufferAllocator, boolean isNested)
protected void	ensurePositionsCapacity(int capacity)
protected static int	getEstimatedBufferMaxCapacity(double targetBufferSize, int unitSize, int positionSize)
protected long	getNullsBufferSerializedSizeInBytes()
protected int[]	getPositions()
protected com.facebook.presto.operator.repartition.DecodedBlockNode	mapPositionsToNestedBlock(com.facebook.presto.operator.repartition.DecodedBlockNode decodedBlockNode) Map the positions for DictionaryBlock to its nested dictionaryBlock, and positions for RunLengthEncodedBlock to its nested value block.
void	noMoreBatches() Signals that this is the last batch in this page, so that the internal buffers in BlockEncodingBuffers can recycled.
protected void	resetNullsBuffer()
protected void	resetPositions()
protected void	serializeNullsTo(io.airlift.slice.SliceOutput output)
protected void	setEstimatedNullsBufferMaxCapacity(int estimatedNullsBufferMaxCapacity)
void	setNextBatch(int positionsOffset, int batchSize) Set the position offset and batch size on the positions array in the BlockEncodingBuffer.
protected abstract void	setupDecodedBlockAndMapPositions(com.facebook.presto.operator.repartition.DecodedBlockNode decodedBlockNode, int partitionBufferCapacity, double decodedBlockPageSizeFraction)
void	setupDecodedBlocksAndPositions(com.facebook.presto.operator.repartition.DecodedBlockNode decodedBlockNode, int[] positions, int positionCount, int partitionBufferCapacity, long estimatedSerializedPageSize) Pass in the decoded block and positions in this block to copy.
String	toString()
protected static void	writeLengthPrefixedString(io.airlift.slice.SliceOutput output, String value)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface com.facebook.presto.operator.repartition.BlockEncodingBuffer

accumulateSerializedRowSizes, appendDataInBatch, getRetainedSizeInBytes, getSerializedSizeInBytes, resetBuffers, serializeTo

Field Detail

GRACE_FACTOR_FOR_MAX_BUFFER_CAPACITY

public static final double GRACE_FACTOR_FOR_MAX_BUFFER_CAPACITY

See Also:

Constant Field Values

bufferAllocator

protected final ArrayAllocator bufferAllocator

isNested

protected final boolean isNested

decodedBlock

protected Block decodedBlock

positionCount

protected int positionCount

batchSize

protected int batchSize

positionsOffset

protected int positionsOffset

bufferedPositionCount

protected int bufferedPositionCount

positionsMapped

protected boolean positionsMapped

flushed

protected boolean flushed

Constructor Detail

AbstractBlockEncodingBuffer

protected AbstractBlockEncodingBuffer(ArrayAllocator bufferAllocator,
                                      boolean isNested)

Method Detail

setupDecodedBlockAndMapPositions

protected abstract void setupDecodedBlockAndMapPositions(com.facebook.presto.operator.repartition.DecodedBlockNode decodedBlockNode,
                                                         int partitionBufferCapacity,
                                                         double decodedBlockPageSizeFraction)

createBlockEncodingBuffers

public static BlockEncodingBuffer createBlockEncodingBuffers(com.facebook.presto.operator.repartition.DecodedBlockNode decodedBlockNode,
                                                             ArrayAllocator bufferAllocator,
                                                             boolean isNested)

setupDecodedBlocksAndPositions

public void setupDecodedBlocksAndPositions(com.facebook.presto.operator.repartition.DecodedBlockNode decodedBlockNode,
                                           int[] positions,
                                           int positionCount,
                                           int partitionBufferCapacity,
                                           long estimatedSerializedPageSize)

Description copied from interface: BlockEncodingBuffer

Pass in the decoded block and positions in this block to copy. Called when a new page is being processed.

Specified by:

setupDecodedBlocksAndPositions in interface BlockEncodingBuffer

setNextBatch

public void setNextBatch(int positionsOffset,
                         int batchSize)

Description copied from interface: BlockEncodingBuffer

Set the position offset and batch size on the positions array in the BlockEncodingBuffer. Called before the next batch is being copied

Specified by:

setNextBatch in interface BlockEncodingBuffer

toString

public String toString()

Overrides:

toString in class Object

setEstimatedNullsBufferMaxCapacity

protected void setEstimatedNullsBufferMaxCapacity(int estimatedNullsBufferMaxCapacity)

getEstimatedBufferMaxCapacity

protected static int getEstimatedBufferMaxCapacity(double targetBufferSize,
                                                   int unitSize,
                                                   int positionSize)

mapPositionsToNestedBlock

protected com.facebook.presto.operator.repartition.DecodedBlockNode mapPositionsToNestedBlock(com.facebook.presto.operator.repartition.DecodedBlockNode decodedBlockNode)

Map the positions for DictionaryBlock to its nested dictionaryBlock, and positions for RunLengthEncodedBlock to its nested value block. For example, positions [0, 2, 5] on DictionaryBlock with dictionary block ['a', 'b', 'c'] and ids [1, 1, 2, 0, 2, 1] will be mapped to [1, 2, 1], and the copied data will be ['b', 'c', 'b'].

ensurePositionsCapacity

protected void ensurePositionsCapacity(int capacity)

appendPositionRange

protected void appendPositionRange(int offset,
                                   int length)

accumulateSerializedRowSizes

protected abstract void accumulateSerializedRowSizes(int[] positionOffsets,
                                                     int positionCount,
                                                     int[] serializedRowSizes)

Called by accumulateSerializedRowSizes(int[] serializedRowSizes) for composite types to add row sizes for nested blocks.

For example, a AbstractBlockEncodingBuffer for an array(int) column with 3 rows - [{1, 2}, {3}, {4, 5, 6}] - will call accumulateSerializedRowSizes with positionOffsets = [0, 2, 3, 6] and serializedRowSizes = [5, 5, 5] (5 = 4 + 1 where 4 bytes for offset and 1 for null flag). After the method returns, serializedRowSizes will be [15, 10, 20].

Parameters:

positionOffsets - The offsets of the top level rows

positionCount - Number of top level rows

resetPositions

protected void resetPositions()

getPositions

protected int[] getPositions()

appendNulls

protected void appendNulls()

noMoreBatches

public void noMoreBatches()

Description copied from interface: BlockEncodingBuffer

Signals that this is the last batch in this page, so that the internal buffers in BlockEncodingBuffers can recycled.

Specified by:

noMoreBatches in interface BlockEncodingBuffer

serializeNullsTo

protected void serializeNullsTo(io.airlift.slice.SliceOutput output)

writeLengthPrefixedString

protected static void writeLengthPrefixedString(io.airlift.slice.SliceOutput output,
                                                String value)

resetNullsBuffer

protected void resetNullsBuffer()

getNullsBufferSerializedSizeInBytes

protected long getNullsBufferSerializedSizeInBytes()