public abstract class RandomAccessDataset extends java.lang.Object implements Dataset
Modifier and Type | Class and Description |
---|---|
static class |
RandomAccessDataset.BaseBuilder<T extends RandomAccessDataset.BaseBuilder<T>>
The Builder to construct a
RandomAccessDataset . |
Dataset.Usage
Modifier and Type | Field and Description |
---|---|
protected Batchifier |
dataBatchifier |
protected Device |
device |
protected java.util.concurrent.ExecutorService |
executor |
protected Batchifier |
labelBatchifier |
protected long |
limit |
protected Pipeline |
pipeline |
protected int |
prefetchNumber |
protected Sampler |
sampler |
protected Pipeline |
targetPipeline |
Constructor and Description |
---|
RandomAccessDataset(RandomAccessDataset.BaseBuilder<?> builder)
Creates a new instance of
RandomAccessDataset with the given necessary
configurations. |
Modifier and Type | Method and Description |
---|---|
protected abstract long |
availableSize()
Returns the number of records available to be read in this
Dataset . |
abstract Record |
get(NDManager manager,
long index)
Gets the
Record for the given index from the dataset. |
java.lang.Iterable<Batch> |
getData(NDManager manager)
Fetches an iterator that can iterate through the
Dataset . |
java.lang.Iterable<Batch> |
getData(NDManager manager,
Sampler sampler)
Fetches an iterator that can iterate through the
Dataset with a custom sampler. |
RandomAccessDataset[] |
randomSplit(int... ratio)
Splits the dataset set into multiple portions.
|
long |
size()
Returns the size of this
Dataset . |
RandomAccessDataset |
subDataset(int fromIndex,
int toIndex)
Returns a view of the portion of this data between the specified
fromIndex ,
inclusive, and toIndex , exclusive. |
ai.djl.util.Pair<java.lang.Number[][],java.lang.Number[][]> |
toArray()
Returns the dataset contents as a Java array.
|
protected Sampler sampler
protected Batchifier dataBatchifier
protected Batchifier labelBatchifier
protected Pipeline pipeline
protected Pipeline targetPipeline
protected java.util.concurrent.ExecutorService executor
protected int prefetchNumber
protected long limit
protected Device device
public RandomAccessDataset(RandomAccessDataset.BaseBuilder<?> builder)
RandomAccessDataset
with the given necessary
configurations.builder
- a builder with the necessary configurationspublic abstract Record get(NDManager manager, long index) throws java.io.IOException
Record
for the given index from the dataset.manager
- the manager used to create the arraysindex
- the index of the requested data itemRecord
that contains the data and label of the requested data itemjava.io.IOException
- if an I/O error occurspublic java.lang.Iterable<Batch> getData(NDManager manager) throws java.io.IOException, TranslateException
Dataset
.getData
in interface Dataset
manager
- the dataset to iterate throughIterable
of Batch
that contains batches of data from the datasetjava.io.IOException
- for various exceptions depending on the datasetTranslateException
- if there is an error while processing inputpublic java.lang.Iterable<Batch> getData(NDManager manager, Sampler sampler) throws java.io.IOException, TranslateException
Dataset
with a custom sampler.manager
- the dataset to iterate throughsampler
- the sampler to use to iterate through the datasetIterable
of Batch
that contains batches of data from the datasetjava.io.IOException
- for various exceptions depending on the datasetTranslateException
- if there is an error while processing inputpublic long size()
Dataset
.Dataset
protected abstract long availableSize()
Dataset
.Dataset
public RandomAccessDataset[] randomSplit(int... ratio) throws java.io.IOException, TranslateException
ratio
- the ratio of each sub datasetjava.io.IOException
- for various exceptions depending on the datasetTranslateException
- if there is an error while processing inputpublic RandomAccessDataset subDataset(int fromIndex, int toIndex)
fromIndex
,
inclusive, and toIndex
, exclusive.fromIndex
- low endpoint (inclusive) of the subDatasettoIndex
- high endpoint (exclusive) of the subDatapublic ai.djl.util.Pair<java.lang.Number[][],java.lang.Number[][]> toArray() throws java.io.IOException, TranslateException
Each Number[] is a flattened dataset record and the Number[][] is the array of all records.
java.io.IOException
- for various exceptions depending on the datasetTranslateException
- if there is an error while processing input