Package org.opensearch.index.store.remote.file

Class OnDemandBlockSnapshotIndexInput

java.lang.Object

org.apache.lucene.store.DataInput

org.apache.lucene.store.IndexInput

org.opensearch.index.store.remote.file.OnDemandBlockSnapshotIndexInput

All Implemented Interfaces:

Closeable, AutoCloseable, Cloneable, org.apache.lucene.store.RandomAccessInput

public class OnDemandBlockSnapshotIndexInput
extends org.apache.lucene.store.IndexInput

This is an implementation of OnDemandBlockIndexInput where this class provides the main IndexInput using shard snapshot files.
This class rely on TransferManager to really fetch the snapshot files from the remote blob store and maybe cache them

Opensearch.internal:

Field Summary

Fields

Modifier and Type	Field	Description
protected final int	blockMask
protected final int	blockSize
protected final int	blockSizeShift	Variables used for block calculation and fetching.
static final String	CLEANER_THREAD_NAME_PREFIX
protected final org.apache.lucene.store.FSDirectory	directory	Underlying lucene directory to open blocks and for caching
protected final BlobStoreIndexShardSnapshot.FileInfo	fileInfo	FileInfo contains snapshot metadata references for this IndexInput
protected final String	fileName	File Name
protected final boolean	isClone	Whether this index input is a clone or otherwise the root file before slicing
protected final long	length	Length of the virtual file, smaller than actual file size if it's a slice
protected final long	offset	Start offset of the virtual file : non-zero in the slice case
protected final long	originalFileSize	Size of the file, larger than length if it's a slice
protected final long	partSize	Maximum size in bytes of snapshot file parts.

Constructor Summary

Constructors

Constructor	Description
OnDemandBlockSnapshotIndexInput(String resourceDescription, BlobStoreIndexShardSnapshot.FileInfo fileInfo, long offset, long length, boolean isClone, org.apache.lucene.store.FSDirectory directory, TransferManager transferManager)
OnDemandBlockSnapshotIndexInput(BlobStoreIndexShardSnapshot.FileInfo fileInfo, org.apache.lucene.store.FSDirectory directory, TransferManager transferManager)

Method Summary

Modifier and Type	Method	Description
static org.opensearch.index.store.remote.file.OnDemandBlockIndexInput.Builder	builder()
protected OnDemandBlockSnapshotIndexInput	buildSlice(String sliceDescription, long offset, long length)	Builds the actual sliced IndexInput (may apply extra offset in subclasses).
OnDemandBlockSnapshotIndexInput	clone()
protected void	cloneBlock(org.opensearch.index.store.remote.file.OnDemandBlockIndexInput other)
void	close()
protected int	currentBlockPosition()
protected long	currentBlockStart()
protected org.apache.lucene.store.IndexInput	fetchBlock(int blockId)	Given a blockId, fetch it's IndexInput which might be partial/split/cloned one
protected long	getActualBlockSize(int blockId)
protected List<BlobFetchRequest.BlobPart>	getBlobParts(long blockStart, long blockEnd)	Returns list of blob parts/chunks in a file for a given block.
protected int	getBlock(long pos)
protected int	getBlockOffset(long pos)
protected long	getBlockStart(int blockId)
long	getFilePointer()
long	length()
byte	readByte()
final byte	readByte(long pos)
final void	readBytes(byte[] b, int offset, int len)
int	readInt()
int	readInt(long pos)
long	readLong()
long	readLong(long pos)
short	readShort()
short	readShort(long pos)
final int	readVInt()
final long	readVLong()
void	seek(long pos)
org.apache.lucene.store.IndexInput	slice(String sliceDescription, long offset, long length)

Methods inherited from class org.apache.lucene.store.IndexInput

getFullSliceDescription, randomAccessSlice, skipBytes, toString

Methods inherited from class org.apache.lucene.store.DataInput

readBytes, readFloats, readGroupVInt, readGroupVInts, readInts, readLongs, readMapOfStrings, readSetOfStrings, readString, readZInt, readZLong

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Details

fileInfo

protected final BlobStoreIndexShardSnapshot.FileInfo fileInfo

FileInfo contains snapshot metadata references for this IndexInput

directory

protected final org.apache.lucene.store.FSDirectory directory

Underlying lucene directory to open blocks and for caching

fileName

protected final String fileName

File Name

partSize

protected final long partSize

Maximum size in bytes of snapshot file parts.

originalFileSize

protected final long originalFileSize

Size of the file, larger than length if it's a slice

CLEANER_THREAD_NAME_PREFIX

public static final String CLEANER_THREAD_NAME_PREFIX

See Also:

• Constant Field Values

offset

protected final long offset

Start offset of the virtual file : non-zero in the slice case

length

protected final long length

Length of the virtual file, smaller than actual file size if it's a slice

isClone

protected final boolean isClone

Whether this index input is a clone or otherwise the root file before slicing

blockSizeShift

protected final int blockSizeShift

Variables used for block calculation and fetching. blockSize must be a power of two, and is defined as 2^blockShiftSize. blockMask is defined as blockSize - 1 and is used to calculate the offset within a block.

blockSize

protected final int blockSize

blockMask

protected final int blockMask

Constructor Details

OnDemandBlockSnapshotIndexInput

public OnDemandBlockSnapshotIndexInput(BlobStoreIndexShardSnapshot.FileInfo fileInfo,
 org.apache.lucene.store.FSDirectory directory,
 TransferManager transferManager)

OnDemandBlockSnapshotIndexInput

public OnDemandBlockSnapshotIndexInput(String resourceDescription,
 BlobStoreIndexShardSnapshot.FileInfo fileInfo,
 long offset,
 long length,
 boolean isClone,
 org.apache.lucene.store.FSDirectory directory,
 TransferManager transferManager)

Method Details

buildSlice

protected OnDemandBlockSnapshotIndexInput buildSlice(String sliceDescription,
 long offset,
 long length)

Builds the actual sliced IndexInput (may apply extra offset in subclasses).

fetchBlock

protected org.apache.lucene.store.IndexInput fetchBlock(int blockId)
                                                 throws IOException

Given a blockId, fetch it's IndexInput which might be partial/split/cloned one

Parameters:

blockId - to fetch for

Returns:

fetched IndexInput

Throws:

IOException

getBlobParts

protected List<BlobFetchRequest.BlobPart> getBlobParts(long blockStart,
 long blockEnd)

Returns list of blob parts/chunks in a file for a given block.

clone

public OnDemandBlockSnapshotIndexInput clone()

getActualBlockSize

protected long getActualBlockSize(int blockId)

slice

public org.apache.lucene.store.IndexInput slice(String sliceDescription,
 long offset,
 long length)
                                         throws IOException

Specified by:

slice in class org.apache.lucene.store.IndexInput

Throws:

IOException

close

public void close()
           throws IOException

Specified by:

close in interface AutoCloseable

Specified by:

close in interface Closeable

Specified by:

close in class org.apache.lucene.store.IndexInput

Throws:

IOException

getFilePointer

public long getFilePointer()

Specified by:

getFilePointer in class org.apache.lucene.store.IndexInput

length

public long length()

Specified by:

length in interface org.apache.lucene.store.RandomAccessInput

Specified by:

length in class org.apache.lucene.store.IndexInput

readByte

public byte readByte()
              throws IOException

Specified by:

readByte in class org.apache.lucene.store.DataInput

Throws:

IOException

readShort

public short readShort()
                throws IOException

Overrides:

readShort in class org.apache.lucene.store.DataInput

Throws:

IOException

readInt

public int readInt()
            throws IOException

Overrides:

readInt in class org.apache.lucene.store.DataInput

Throws:

IOException

readLong

public long readLong()
              throws IOException

Overrides:

readLong in class org.apache.lucene.store.DataInput

Throws:

IOException

readVInt

public final int readVInt()
                   throws IOException

Overrides:

readVInt in class org.apache.lucene.store.DataInput

Throws:

IOException

readVLong

public final long readVLong()
                     throws IOException

Overrides:

readVLong in class org.apache.lucene.store.DataInput

Throws:

IOException

seek

public void seek(long pos)
          throws IOException

Specified by:

seek in class org.apache.lucene.store.IndexInput

Throws:

IOException

readByte

public final byte readByte(long pos)
                    throws IOException

Specified by:

readByte in interface org.apache.lucene.store.RandomAccessInput

Throws:

IOException

readShort

public short readShort(long pos)
                throws IOException

Specified by:

readShort in interface org.apache.lucene.store.RandomAccessInput

Throws:

IOException

readInt

public int readInt(long pos)
            throws IOException

Specified by:

readInt in interface org.apache.lucene.store.RandomAccessInput

Throws:

IOException

readLong

public long readLong(long pos)
              throws IOException

Specified by:

readLong in interface org.apache.lucene.store.RandomAccessInput

Throws:

IOException

readBytes

public final void readBytes(byte[] b,
 int offset,
 int len)
                     throws IOException

Specified by:

readBytes in class org.apache.lucene.store.DataInput

Throws:

IOException

cloneBlock

protected void cloneBlock(org.opensearch.index.store.remote.file.OnDemandBlockIndexInput other)

getBlock

protected int getBlock(long pos)

getBlockOffset

protected int getBlockOffset(long pos)

getBlockStart

protected long getBlockStart(int blockId)

currentBlockStart

protected long currentBlockStart()

currentBlockPosition

protected int currentBlockPosition()

builder

public static org.opensearch.index.store.remote.file.OnDemandBlockIndexInput.Builder builder()