Package ai.djl.training.dataset
Interface Dataset
-
- All Known Subinterfaces:
RawDataset<T>
- All Known Implementing Classes:
ArrayDataset
,RandomAccessDataset
public interface Dataset
An interface to represent a set of sample data/label pairs to train a model.
-
-
Nested Class Summary
Nested Classes Modifier and Type Interface Description static class
Dataset.Usage
An enum that indicates the mode - training, test or validation.
-
Method Summary
All Methods Instance Methods Abstract Methods Default Methods Modifier and Type Method Description java.lang.Iterable<Batch>
getData(NDManager manager)
Fetches an iterator that can iterate through theDataset
.default java.lang.Iterable<Batch>
getData(NDManager manager, java.util.concurrent.ExecutorService executorService)
Fetches an iterator that can iterate through theDataset
with multiple threads.default TranslatorOptions
matchingTranslatorOptions()
ReturnsTranslatorOptions
that match the pre-processing and post-processing of this dataset.default void
prepare()
Prepares the dataset for use.void
prepare(ai.djl.util.Progress progress)
Prepares the dataset for use with tracked progress.
-
-
-
Method Detail
-
getData
java.lang.Iterable<Batch> getData(NDManager manager) throws java.io.IOException, TranslateException
Fetches an iterator that can iterate through theDataset
.- Parameters:
manager
- the dataset to iterate through- Returns:
- an
Iterable
ofBatch
that contains batches of data from the dataset - Throws:
java.io.IOException
- for various exceptions depending on the datasetTranslateException
- if there is an error while processing input
-
getData
default java.lang.Iterable<Batch> getData(NDManager manager, java.util.concurrent.ExecutorService executorService) throws java.io.IOException, TranslateException
Fetches an iterator that can iterate through theDataset
with multiple threads.- Parameters:
manager
- the dataset to iterate throughexecutorService
- the executorService to use for multi-threading- Returns:
- an
Iterable
ofBatch
that contains batches of data from the dataset - Throws:
java.io.IOException
- for various exceptions depending on the datasetTranslateException
- if there is an error while processing input
-
prepare
default void prepare() throws java.io.IOException, TranslateException
Prepares the dataset for use.- Throws:
java.io.IOException
- for various exceptions depending on the datasetTranslateException
- if there is an error while processing input
-
prepare
void prepare(ai.djl.util.Progress progress) throws java.io.IOException, TranslateException
Prepares the dataset for use with tracked progress.- Parameters:
progress
- the progress tracker- Throws:
java.io.IOException
- for various exceptions depending on the datasetTranslateException
- if there is an error while processing input
-
matchingTranslatorOptions
default TranslatorOptions matchingTranslatorOptions()
ReturnsTranslatorOptions
that match the pre-processing and post-processing of this dataset.- Returns:
- matching translators or null if none defined
-
-