Package org.apache.druid.frame
Class Frame
- java.lang.Object
-
- org.apache.druid.frame.Frame
-
public class Frame extends Object
A data frame. Frames are split into contiguous "regions". With columnar frames (FrameType.COLUMNAR
) each region is a column. With row-based frames (FrameType.ROW_BASED
) there are always two regions: row offsets and row data. This object is lightweight. It has constant overhead regardless of the number of rows or regions. Frames wrap someMemory
. If the memory is backed by a resource that requires explicit releasing, such as direct off-heap memory or a memory-mapped file, the creator of the Memory is responsible for releasing that resource when the frame is no longer needed. Frames are written withFrameWriter
and read withFrameReader
. Frame format: - 1 byte:FrameType.version()
- 8 bytes: size in bytes of the frame, little-endian long - 4 bytes: number of rows, little-endian int - 4 bytes: number of regions, little-endian int - 1 byte: 0 if frame is nonpermuted, 1 if frame is permuted - 4 bytes x numRows: permutation section; only present for permuted frames. Array of little-endian ints mapping logical row numbers to physical row numbers. - 8 bytes x numRegions: region offsets. Array of end offsets of each region (exclusive), relative to start of frame, as little-endian longs. - NNN bytes: regions, back-to-back. There is also a compressed frame format. Compressed frames are written bywriteTo(java.nio.channels.WritableByteChannel, boolean, java.nio.ByteBuffer, org.apache.druid.frame.channel.ByteTracker)
when "compress" is true, and decompressed bydecompress(org.apache.datasketches.memory.Memory, long, long)
. Format: - 1 byte: compression type:CompressionStrategy.getId()
. Currently, only LZ4 is supported. - 8 bytes: compressed frame length, little-endian long - 8 bytes: uncompressed frame length (numBytes), little-endian long - NNN bytes: LZ4-compressed frame - 8 bytes: 64-bit xxhash checksum of prior content, including 16-byte header and compressed frame, little-endian long Note to developers: if we end up needing to add more fields here, consider introducing a Smile (or Protobuf, etc) header to make it simpler to add more fields.
-
-
Field Summary
Fields Modifier and Type Field Description static int
COMPRESSED_FRAME_ENVELOPE_SIZE
static int
COMPRESSED_FRAME_HEADER_SIZE
static int
COMPRESSED_FRAME_TRAILER_SIZE
static long
HEADER_SIZE
-
Method Summary
All Methods Static Methods Instance Methods Concrete Methods Modifier and Type Method Description static int
compressionBufferSize(long frameBytes)
Minimum size of compression buffer required bywriteTo(java.nio.channels.WritableByteChannel, boolean, java.nio.ByteBuffer, org.apache.druid.frame.channel.ByteTracker)
when "compress" is true.static Frame
decompress(org.apache.datasketches.memory.Memory memory, long position, long length)
Decompresses the provided memory and returns a frame backed by that decompressed memory.boolean
isPermuted()
long
numBytes()
int
numRegions()
int
numRows()
int
physicalRow(int logicalRow)
Maps a logical row number to a physical row number.org.apache.datasketches.memory.Memory
region(int regionNumber)
Returns memory corresponding to a particular region of this frame.FrameType
type()
static Frame
wrap(byte[] bytes)
Returns a frame backed by the provided byte array.static Frame
wrap(ByteBuffer buffer)
Returns a frame backed by the provided ByteBuffer.static Frame
wrap(org.apache.datasketches.memory.Memory memory)
Returns a frame backed by the provided Memory.org.apache.datasketches.memory.WritableMemory
writableMemory()
Direct, writable access to this frame's memory.long
writeTo(WritableByteChannel channel, boolean compress, ByteBuffer compressionBuffer, ByteTracker byteTracker)
Writes this frame to a channel, optionally compressing it as well.
-
-
-
Field Detail
-
HEADER_SIZE
public static final long HEADER_SIZE
- See Also:
- Constant Field Values
-
COMPRESSED_FRAME_HEADER_SIZE
public static final int COMPRESSED_FRAME_HEADER_SIZE
- See Also:
- Constant Field Values
-
COMPRESSED_FRAME_TRAILER_SIZE
public static final int COMPRESSED_FRAME_TRAILER_SIZE
- See Also:
- Constant Field Values
-
COMPRESSED_FRAME_ENVELOPE_SIZE
public static final int COMPRESSED_FRAME_ENVELOPE_SIZE
- See Also:
- Constant Field Values
-
-
Method Detail
-
wrap
public static Frame wrap(org.apache.datasketches.memory.Memory memory)
Returns a frame backed by the provided Memory. This operation does not do any copies or allocations. The Memory must be in little-endian byte order. Behavior is undefined if the memory is modified anytime during the lifetime of the Frame object.
-
wrap
public static Frame wrap(ByteBuffer buffer)
Returns a frame backed by the provided ByteBuffer. This operation does not do any copies or allocations. The position and limit of the buffer are ignored. If you need them to be respected, callByteBuffer.slice()
first, or usewrap(Memory)
to wrap a particular region.
-
wrap
public static Frame wrap(byte[] bytes)
Returns a frame backed by the provided byte array. This operation does not do any copies or allocations. The position and limit of the buffer are ignored. If you need them to be respected, callByteBuffer.slice()
first, or usewrap(Memory)
to wrap a particular region.
-
decompress
public static Frame decompress(org.apache.datasketches.memory.Memory memory, long position, long length)
Decompresses the provided memory and returns a frame backed by that decompressed memory. This operation is safe even on corrupt data: it validates position, length, and checksum prior to decompressing. This operation allocates memory on-heap to store the decompressed frame.
-
compressionBufferSize
public static int compressionBufferSize(long frameBytes)
Minimum size of compression buffer required bywriteTo(java.nio.channels.WritableByteChannel, boolean, java.nio.ByteBuffer, org.apache.druid.frame.channel.ByteTracker)
when "compress" is true.
-
type
public FrameType type()
-
numBytes
public long numBytes()
-
numRows
public int numRows()
-
numRegions
public int numRegions()
-
isPermuted
public boolean isPermuted()
-
physicalRow
public int physicalRow(int logicalRow)
Maps a logical row number to a physical row number. If the frame is non-permuted, these are the same. If the frame is permuted, this uses the sorted-row mappings to remap the row number.- Throws:
IllegalArgumentException
- if "logicalRow" is out of bounds
-
region
public org.apache.datasketches.memory.Memory region(int regionNumber)
Returns memory corresponding to a particular region of this frame.
-
writableMemory
public org.apache.datasketches.memory.WritableMemory writableMemory()
Direct, writable access to this frame's memory. Used by operations that modify the frame in-place, likeFrameSort
. Most callers should useregion(int)
andphysicalRow(int)
, rather than this direct-access method.- Throws:
IllegalStateException
- if this frame wraps non-writable memory
-
writeTo
public long writeTo(WritableByteChannel channel, boolean compress, @Nullable ByteBuffer compressionBuffer, ByteTracker byteTracker) throws IOException
Writes this frame to a channel, optionally compressing it as well. Returns the number of bytes written. The providedcompressionBuffer
is used to hold compressed data temporarily, prior to writing it to the channel. It must be at least as large ascompressionBufferSize(numBytes())
, or else anIllegalStateException
is thrown. It may be null if "compress" is false.- Throws:
IOException
-
-