public abstract class RandomAccessDataset extends java.lang.Object implements Dataset, java.util.RandomAccess
Modifier and Type | Class and Description |
---|---|
static class |
RandomAccessDataset.BaseBuilder<T extends RandomAccessDataset.BaseBuilder>
The Builder to construct a
RandomAccessDataset . |
Dataset.Usage
Modifier and Type | Field and Description |
---|---|
protected Batchifier |
dataBatchifier |
protected Device |
device |
protected java.util.concurrent.ExecutorService |
executor |
protected Batchifier |
labelBatchifier |
protected long |
limit |
protected Pipeline |
pipeline |
protected int |
prefetchNumber |
protected Sampler |
sampler |
protected Pipeline |
targetPipeline |
Constructor and Description |
---|
RandomAccessDataset(RandomAccessDataset.BaseBuilder<?> builder)
Creates a new instance of
RandomAccessDataset with the given necessary
configurations. |
Modifier and Type | Method and Description |
---|---|
protected abstract long |
availableSize()
Returns the number of records available to be read in this
Dataset . |
abstract Record |
get(NDManager manager,
long index)
Gets the
Record for the given index from the dataset. |
java.lang.Iterable<Batch> |
getData(NDManager manager)
Fetches an iterator that can iterate through the
Dataset . |
java.lang.Iterable<Batch> |
getData(NDManager manager,
Sampler sampler)
Fetches an iterator that can iterate through the
Dataset with a custom sampler. |
RandomAccessDataset[] |
randomSplit(int... ratio)
Splits the dataset set into multiple portions.
|
long |
size()
Returns the size of this
Dataset . |
protected Sampler sampler
protected Batchifier dataBatchifier
protected Batchifier labelBatchifier
protected Pipeline pipeline
protected Pipeline targetPipeline
protected java.util.concurrent.ExecutorService executor
protected int prefetchNumber
protected long limit
protected Device device
public RandomAccessDataset(RandomAccessDataset.BaseBuilder<?> builder)
RandomAccessDataset
with the given necessary
configurations.builder
- a builder with the necessary configurationspublic abstract Record get(NDManager manager, long index) throws java.io.IOException, TranslateException
Record
for the given index from the dataset.manager
- the manager used to create the arraysindex
- the index of the requested data itemRecord
that contains the data and label of the requested data itemjava.io.IOException
- if an I/O error occursTranslateException
- if there is an error while processing inputpublic java.lang.Iterable<Batch> getData(NDManager manager)
Dataset
.public java.lang.Iterable<Batch> getData(NDManager manager, Sampler sampler)
Dataset
with a custom sampler.manager
- the dataset to iterate throughsampler
- the sampler to use to iterate through the datasetIterable
of Batch
that contains batches of data from the datasetpublic long size()
Dataset
.Dataset
protected abstract long availableSize()
Dataset
.Dataset
public RandomAccessDataset[] randomSplit(int... ratio)
ratio
- the ratio of each sub dataset