org.nd4j.linalg.dataset

Class AsyncDataSetIterator

java.lang.Object

org.nd4j.linalg.dataset.AsyncDataSetIterator

All Implemented Interfaces:

Serializable, Iterator<DataSet>, DataSetIterator

public class AsyncDataSetIterator
extends Object
implements DataSetIterator

Async prefetching iterator wrapper for DataSetIterator implementations. This will asynchronously prefetch the specified number of minibatches from the underlying iterator.
Also has the option (enabled by default for most constructors) to use a cyclical workspace to avoid creating INDArrays with off-heap memory that needs to be cleaned up by the JVM garbage collector.
Note that appropriate DL4J fit methods automatically utilize this iterator, so users don't need to manually wrap their iterators when fitting a network

Author:

[email protected]

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected class	AsyncDataSetIterator.AsyncPrefetchThread

Field Summary

Fields

Modifier and Type	Field and Description
protected DataSetIterator	backedIterator
protected BlockingQueue<DataSet>	buffer
protected DataSetCallback	callback
protected Integer	deviceId
protected AtomicBoolean	hasDepleted
protected DataSet	nextElement
protected int	prefetchSize
protected AtomicBoolean	shouldWork
protected DataSet	terminator
protected AsyncDataSetIterator.AsyncPrefetchThread	thread
protected RuntimeException	throwable
protected boolean	useWorkspace
protected String	workspaceId

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	AsyncDataSetIterator()
	AsyncDataSetIterator(DataSetIterator baseIterator) Create an Async iterator with the default queue size of 8
	AsyncDataSetIterator(DataSetIterator baseIterator, int queueSize)
	AsyncDataSetIterator(DataSetIterator iterator, int queueSize, BlockingQueue<DataSet> queue) Create an Async iterator with the default queue size of 8
	AsyncDataSetIterator(DataSetIterator iterator, int queueSize, BlockingQueue<DataSet> queue, boolean useWorkspace)
	AsyncDataSetIterator(DataSetIterator iterator, int queueSize, BlockingQueue<DataSet> queue, boolean useWorkspace, DataSetCallback callback)
	AsyncDataSetIterator(DataSetIterator iterator, int queueSize, BlockingQueue<DataSet> queue, boolean useWorkspace, DataSetCallback callback, Integer deviceId)
	AsyncDataSetIterator(DataSetIterator baseIterator, int queueSize, boolean useWorkspace)
	AsyncDataSetIterator(DataSetIterator baseIterator, int queueSize, boolean useWorkspace, DataSetCallback callback)
	AsyncDataSetIterator(DataSetIterator baseIterator, int queueSize, boolean useWorkspace, Integer deviceId)

Method Summary

Modifier and Type	Method and Description
boolean	asyncSupported() Does this DataSetIterator support asynchronous prefetching of multiple DataSet objects? Most DataSetIterators do, but in some cases it may not make sense to wrap this iterator in an iterator that does asynchronous prefetching.
int	batch() Batch size
protected void	externalCall()
List<String>	getLabels() Get dataset iterator record reader labels
DataSetPreProcessor	getPreProcessor() Returns preprocessors, if defined
boolean	hasNext() Returns true if the iteration has more elements.
int	inputColumns() Input columns for the dataset
DataSet	next() Returns the next element in the iteration.
DataSet	next(int num) Like the standard next method but allows a customizable number of examples returned
void	remove() Removes from the underlying collection the last element returned by this iterator (optional operation).
void	reset() Resets the iterator back to the beginning
boolean	resetSupported() Is resetting supported by this DataSetIterator? Many DataSetIterators do support resetting, but some don't
void	setPreProcessor(DataSetPreProcessor preProcessor) Set a pre processor
void	shutdown() This method will terminate background thread AND will destroy attached workspace (if any) PLEASE NOTE: After shutdown() call, this instance can't be used anymore
int	totalOutcomes() The number of labels for the dataset

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface java.util.Iterator

forEachRemaining

Field Detail

backedIterator

protected DataSetIterator backedIterator

terminator

protected DataSet terminator

nextElement

protected DataSet nextElement

buffer

protected BlockingQueue<DataSet> buffer

thread

protected AsyncDataSetIterator.AsyncPrefetchThread thread

shouldWork

protected AtomicBoolean shouldWork

throwable

protected volatile RuntimeException throwable

useWorkspace

protected boolean useWorkspace

prefetchSize

protected int prefetchSize

workspaceId

protected String workspaceId

deviceId

protected Integer deviceId

hasDepleted

protected AtomicBoolean hasDepleted

callback

protected DataSetCallback callback

Constructor Detail

AsyncDataSetIterator

protected AsyncDataSetIterator()

AsyncDataSetIterator

public AsyncDataSetIterator(DataSetIterator baseIterator)

Create an Async iterator with the default queue size of 8

Parameters:

baseIterator - Underlying iterator to wrap and fetch asynchronously from

AsyncDataSetIterator

public AsyncDataSetIterator(DataSetIterator iterator,
                            int queueSize,
                            BlockingQueue<DataSet> queue)

Create an Async iterator with the default queue size of 8

Parameters:

iterator - Underlying iterator to wrap and fetch asynchronously from

queue - Queue size - number of iterators to

AsyncDataSetIterator

public AsyncDataSetIterator(DataSetIterator baseIterator,
                            int queueSize)

AsyncDataSetIterator

public AsyncDataSetIterator(DataSetIterator baseIterator,
                            int queueSize,
                            boolean useWorkspace)

AsyncDataSetIterator

public AsyncDataSetIterator(DataSetIterator baseIterator,
                            int queueSize,
                            boolean useWorkspace,
                            Integer deviceId)

AsyncDataSetIterator

public AsyncDataSetIterator(DataSetIterator baseIterator,
                            int queueSize,
                            boolean useWorkspace,
                            DataSetCallback callback)

AsyncDataSetIterator

public AsyncDataSetIterator(DataSetIterator iterator,
                            int queueSize,
                            BlockingQueue<DataSet> queue,
                            boolean useWorkspace)

AsyncDataSetIterator

public AsyncDataSetIterator(DataSetIterator iterator,
                            int queueSize,
                            BlockingQueue<DataSet> queue,
                            boolean useWorkspace,
                            DataSetCallback callback)

AsyncDataSetIterator

public AsyncDataSetIterator(DataSetIterator iterator,
                            int queueSize,
                            BlockingQueue<DataSet> queue,
                            boolean useWorkspace,
                            DataSetCallback callback,
                            Integer deviceId)

Method Detail

next

public DataSet next(int num)

Like the standard next method but allows a customizable number of examples returned

Specified by:

next in interface DataSetIterator

Parameters:

num - the number of examples

Returns:

the next data applyTransformToDestination

inputColumns

public int inputColumns()

Input columns for the dataset

Specified by:

inputColumns in interface DataSetIterator

Returns:

totalOutcomes

public int totalOutcomes()

The number of labels for the dataset

Specified by:

totalOutcomes in interface DataSetIterator

Returns:

resetSupported

public boolean resetSupported()

Is resetting supported by this DataSetIterator? Many DataSetIterators do support resetting, but some don't

Specified by:

resetSupported in interface DataSetIterator

Returns:

true if reset method is supported; false otherwise

asyncSupported

public boolean asyncSupported()

Does this DataSetIterator support asynchronous prefetching of multiple DataSet objects? Most DataSetIterators do, but in some cases it may not make sense to wrap this iterator in an iterator that does asynchronous prefetching. For example, it would not make sense to use asynchronous prefetching for the following types of iterators: (a) Iterators that store their full contents in memory already (b) Iterators that re-use features/labels arrays (as future next() calls will overwrite past contents) (c) Iterators that already implement some level of asynchronous prefetching (d) Iterators that may return different data depending on when the next() method is called

Specified by:

asyncSupported in interface DataSetIterator

Returns:

true if asynchronous prefetching from this iterator is OK; false if asynchronous prefetching should not be used with this iterator

externalCall

protected void externalCall()

reset

public void reset()

Resets the iterator back to the beginning

Specified by:

reset in interface DataSetIterator

shutdown

public void shutdown()

This method will terminate background thread AND will destroy attached workspace (if any) PLEASE NOTE: After shutdown() call, this instance can't be used anymore

batch

public int batch()

Batch size

Specified by:

batch in interface DataSetIterator

Returns:

setPreProcessor

public void setPreProcessor(DataSetPreProcessor preProcessor)

Set a pre processor

Specified by:

setPreProcessor in interface DataSetIterator

Parameters:

preProcessor - a pre processor to set

getPreProcessor

public DataSetPreProcessor getPreProcessor()

Returns preprocessors, if defined

Specified by:

getPreProcessor in interface DataSetIterator

Returns:

getLabels

public List<String> getLabels()

Get dataset iterator record reader labels

Specified by:

getLabels in interface DataSetIterator

hasNext

public boolean hasNext()

Returns true if the iteration has more elements. (In other words, returns true if next(int) would return an element rather than throwing an exception.)

Specified by:

hasNext in interface Iterator<DataSet>

Returns:

true if the iteration has more elements

next

public DataSet next()

Returns the next element in the iteration.

Specified by:

next in interface Iterator<DataSet>

Returns:

the next element in the iteration

remove

public void remove()

Removes from the underlying collection the last element returned by this iterator (optional operation). This method can be called only once per call to next(int). The behavior of an iterator is unspecified if the underlying collection is modified while the iteration is in progress in any way other than by calling this method.

Specified by:

remove in interface Iterator<DataSet>

Throws:

UnsupportedOperationException - if the remove operation is not supported by this iterator

IllegalStateException - if the next method has not yet been called, or the remove method has already been called after the last call to the next method