ai.djl.training.dataset

Interface Dataset

All Known Implementing Classes:

ArrayDataset, RandomAccessDataset

public interface Dataset

An interface to represent a set of sample data/label pairs to train a model.

See Also:

The guide on datasets

Nested Class Summary

Nested Classes

Modifier and Type	Interface and Description
static class	Dataset.Usage An enum that indicates the mode - training, test or validation.

Method Summary

Modifier and Type	Method and Description
java.lang.Iterable<Batch>	getData(NDManager manager) Fetches an iterator that can iterate through the Dataset.
default java.lang.Iterable<Batch>	getData(NDManager manager, java.util.concurrent.ExecutorService executorService) Fetches an iterator that can iterate through the Dataset with multiple threads.
default void	prepare() Prepares the dataset for use.
void	prepare(ai.djl.util.Progress progress) Prepares the dataset for use with tracked progress.

Method Detail

getData

java.lang.Iterable<Batch> getData(NDManager manager)
                           throws java.io.IOException,
                                  TranslateException

Fetches an iterator that can iterate through the Dataset.

Parameters:

manager - the dataset to iterate through

Returns:

an Iterable of Batch that contains batches of data from the dataset

Throws:

java.io.IOException - for various exceptions depending on the dataset

TranslateException - if there is an error while processing input

getData

default java.lang.Iterable<Batch> getData(NDManager manager,
                                          java.util.concurrent.ExecutorService executorService)
                                   throws java.io.IOException,
                                          TranslateException

Fetches an iterator that can iterate through the Dataset with multiple threads.

Parameters:

manager - the dataset to iterate through

executorService - the executorService to use for multi-threading

Returns:

an Iterable of Batch that contains batches of data from the dataset

Throws:

java.io.IOException - for various exceptions depending on the dataset

TranslateException - if there is an error while processing input

prepare

default void prepare()
              throws java.io.IOException,
                     TranslateException

Prepares the dataset for use.

Throws:

java.io.IOException - for various exceptions depending on the dataset

TranslateException - if there is an error while processing input

prepare

void prepare(ai.djl.util.Progress progress)
      throws java.io.IOException,
             TranslateException

Prepares the dataset for use with tracked progress.

Parameters:

progress - the progress tracker

Throws:

java.io.IOException - for various exceptions depending on the dataset

TranslateException - if there is an error while processing input