Package org.apache.druid.storage.remote

Class ChunkingStorageConnector<T>

java.lang.Object

org.apache.druid.storage.remote.ChunkingStorageConnector<T>

All Implemented Interfaces:

StorageConnector

public abstract class ChunkingStorageConnector<T>
extends Object
implements StorageConnector

An abstract implementation of the storage connectors that download the file from the remote storage in chunks and presents the downloaded chunks as a single InputStream for the consumers of the connector. This implementation benefits over keeping the InputStream to the remote source open since we don't require the connection to be open for the entire duration. Checkout ChunkingStorageConnectorParameters to see the inputs required to support chunking

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static interface	ChunkingStorageConnector.GetObjectFromRangeFunction<T>

Constructor Summary

Constructors

Constructor	Description
ChunkingStorageConnector()
ChunkingStorageConnector(long chunkSizeBytes)

Method Summary

Modifier and Type	Method	Description
abstract ChunkingStorageConnectorParameters<T>	buildInputParams(String path)
abstract ChunkingStorageConnectorParameters<T>	buildInputParams(String path, long from, long size)
InputStream	read(String path)	Reads the data present at the path in the underlying storage system.
InputStream	readRange(String path, long from, long size)	Reads the data present for a given range at the path in the underlying storage system.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.druid.storage.StorageConnector

deleteFile, deleteFiles, deleteRecursively, listDir, pathExists, write

Constructor Detail

ChunkingStorageConnector

public ChunkingStorageConnector()

ChunkingStorageConnector

public ChunkingStorageConnector(long chunkSizeBytes)

Method Detail

read

public InputStream read(String path)
                 throws IOException

Description copied from interface: StorageConnector

Reads the data present at the path in the underlying storage system. Most implementations prepend the input path with a basePath. The caller should take care of closing the stream when done or in case of error.

Specified by:

read in interface StorageConnector

Returns:

InputStream

Throws:

IOException - if the path is not present or the unable to read the data present on the path.

readRange

public InputStream readRange(String path,
                             long from,
                             long size)

Description copied from interface: StorageConnector

Reads the data present for a given range at the path in the underlying storage system. Most implementations prepend the input path with a basePath. The caller should take care of closing the stream when done or in case of error. Further, the caller must ensure that the start offset and the size of the read are valid parameters for the given path for correct behavior.

Specified by:

readRange in interface StorageConnector

Parameters:

path - The path to read data from

from - Start offset of the read in the path

size - Length of the read to be done

Returns:

InputStream starting from the given offset limited by the given size

buildInputParams

public abstract ChunkingStorageConnectorParameters<T> buildInputParams(String path)
                                                                throws IOException

Throws:

IOException

buildInputParams

public abstract ChunkingStorageConnectorParameters<T> buildInputParams(String path,
                                                                       long from,
                                                                       long size)