org.nd4j.linalg.dataset.api.iterator

Class KFoldIterator

java.lang.Object

org.nd4j.linalg.dataset.api.iterator.KFoldIterator

All Implemented Interfaces:

Serializable, Iterator<DataSet>, DataSetIterator

public class KFoldIterator
extends Object
implements DataSetIterator

Splits a dataset into k folds. DataSet is duplicated in memory once call .next() to get the k-1 folds to train on and call .testfold() to get the corresponding kth fold for testing

Author:

Susan Eraly

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected DataSetPreProcessor	preProcessor

Constructor Summary

Constructors

Constructor and Description
KFoldIterator(DataSet singleFold)
KFoldIterator(int k, DataSet singleFold) Create an iterator given the dataset and a value of k (optional, defaults to 10) If number of samples in the dataset is not a multiple of k, the last fold will have less samples with the rest having the same number of samples.

Method Summary

Modifier and Type	Method and Description
boolean	asyncSupported() Does this DataSetIterator support asynchronous prefetching of multiple DataSet objects? Most DataSetIterators do, but in some cases it may not make sense to wrap this iterator in an iterator that does asynchronous prefetching.
int	batch() The number of examples in every fold, except the last if totalexamples % k !=0
int	cursor() cursor value of zero indicates no iterations and the very first fold will be held out as test cursor value of 1 indicates the the next() call will return all but second fold which will be held out as test curson value of k-1 indicates the next() call will return all but the last fold which will be held out as test
List<String>	getLabels() Get dataset iterator record reader labels
DataSetPreProcessor	getPreProcessor() Returns preprocessors, if defined
boolean	hasNext()
int	inputColumns() Input columns for the dataset
int	lastBatch() The number of examples in the last fold if totalexamples % k == 0 same as the number of examples in every other fold
DataSet	next()
DataSet	next(int num) Like the standard next method but allows a customizable number of examples returned
int	numExamples() Total number of examples in the dataset
void	remove()
void	reset() Shuffles the dataset and resets to the first fold
boolean	resetSupported() Is resetting supported by this DataSetIterator? Many DataSetIterators do support resetting, but some don't
void	setPreProcessor(DataSetPreProcessor preProcessor) Set a pre processor
DataSet	testFold()
int	totalExamples() Returns total number of examples in the dataset (all k folds)
int	totalOutcomes() The number of labels for the dataset

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface java.util.Iterator

forEachRemaining

Field Detail

preProcessor

protected DataSetPreProcessor preProcessor

Constructor Detail

KFoldIterator

public KFoldIterator(DataSet singleFold)

KFoldIterator

public KFoldIterator(int k,
                     DataSet singleFold)

Create an iterator given the dataset and a value of k (optional, defaults to 10) If number of samples in the dataset is not a multiple of k, the last fold will have less samples with the rest having the same number of samples.

Parameters:

k - number of folds (optional, defaults to 10)

singleFold - DataSet to split into k folds

Method Detail

next

public DataSet next(int num)
             throws UnsupportedOperationException

Description copied from interface: DataSetIterator

Like the standard next method but allows a customizable number of examples returned

Specified by:

next in interface DataSetIterator

Parameters:

num - the number of examples

Returns:

the next data applyTransformToDestination

Throws:

UnsupportedOperationException

totalExamples

public int totalExamples()

Returns total number of examples in the dataset (all k folds)

Specified by:

totalExamples in interface DataSetIterator

Returns:

total number of examples in the dataset including all k folds

inputColumns

public int inputColumns()

Description copied from interface: DataSetIterator

Input columns for the dataset

Specified by:

inputColumns in interface DataSetIterator

Returns:

totalOutcomes

public int totalOutcomes()

Description copied from interface: DataSetIterator

The number of labels for the dataset

Specified by:

totalOutcomes in interface DataSetIterator

Returns:

resetSupported

public boolean resetSupported()

Description copied from interface: DataSetIterator

Is resetting supported by this DataSetIterator? Many DataSetIterators do support resetting, but some don't

Specified by:

resetSupported in interface DataSetIterator

Returns:

true if reset method is supported; false otherwise

asyncSupported

public boolean asyncSupported()

Description copied from interface: DataSetIterator

Does this DataSetIterator support asynchronous prefetching of multiple DataSet objects? Most DataSetIterators do, but in some cases it may not make sense to wrap this iterator in an iterator that does asynchronous prefetching. For example, it would not make sense to use asynchronous prefetching for the following types of iterators: (a) Iterators that store their full contents in memory already (b) Iterators that re-use features/labels arrays (as future next() calls will overwrite past contents) (c) Iterators that already implement some level of asynchronous prefetching (d) Iterators that may return different data depending on when the next() method is called

Specified by:

asyncSupported in interface DataSetIterator

Returns:

true if asynchronous prefetching from this iterator is OK; false if asynchronous prefetching should not be used with this iterator

reset

public void reset()

Shuffles the dataset and resets to the first fold

Specified by:

reset in interface DataSetIterator

batch

public int batch()

The number of examples in every fold, except the last if totalexamples % k !=0

Specified by:

batch in interface DataSetIterator

Returns:

examples in a fold

lastBatch

public int lastBatch()

The number of examples in the last fold if totalexamples % k == 0 same as the number of examples in every other fold

Returns:

examples in the last fold

cursor

public int cursor()

cursor value of zero indicates no iterations and the very first fold will be held out as test cursor value of 1 indicates the the next() call will return all but second fold which will be held out as test curson value of k-1 indicates the next() call will return all but the last fold which will be held out as test

Specified by:

cursor in interface DataSetIterator

Returns:

the value of the cursor which indicates which fold is held out as a test set

numExamples

public int numExamples()

Description copied from interface: DataSetIterator

Total number of examples in the dataset

Specified by:

numExamples in interface DataSetIterator

Returns:

setPreProcessor

public void setPreProcessor(DataSetPreProcessor preProcessor)

Description copied from interface: DataSetIterator

Set a pre processor

Specified by:

setPreProcessor in interface DataSetIterator

Parameters:

preProcessor - a pre processor to set

getPreProcessor

public DataSetPreProcessor getPreProcessor()

Description copied from interface: DataSetIterator

Returns preprocessors, if defined

Specified by:

getPreProcessor in interface DataSetIterator

Returns:

getLabels

public List<String> getLabels()

Description copied from interface: DataSetIterator

Get dataset iterator record reader labels

Specified by:

getLabels in interface DataSetIterator

hasNext

public boolean hasNext()

Specified by:

hasNext in interface Iterator<DataSet>

next

public DataSet next()

Specified by:

next in interface Iterator<DataSet>

remove

public void remove()

Specified by:

remove in interface Iterator<DataSet>

testFold

public DataSet testFold()

Returns:

the held out fold as a dataset