Package org.nd4j.linalg.dataset
Class MultiDataSet
- java.lang.Object
-
- org.nd4j.linalg.dataset.MultiDataSet
-
- All Implemented Interfaces:
Serializable
,MultiDataSet
public class MultiDataSet extends Object implements MultiDataSet
- See Also:
- Serialized Form
-
-
Constructor Summary
Constructors Constructor Description MultiDataSet()
Create a new (empty) MultiDataSet object (all fields are null)MultiDataSet(INDArray[] features, INDArray[] labels)
MultiDataSet constructor with no mask arraysMultiDataSet(INDArray[] features, INDArray[] labels, INDArray[] featuresMaskArrays, INDArray[] labelsMaskArrays)
MultiDataSet(INDArray[] features, INDArray[] labels, INDArray[] featuresMaskArrays, INDArray[] labelsMaskArrays, List<Serializable> exampleMetaData)
MultiDataSet(INDArray features, INDArray labels)
MultiDataSet constructor with single features/labels input, no mask arraysMultiDataSet(INDArray features, INDArray labels, INDArray featuresMask, INDArray labelsMask)
MultiDataSet constructor with single features/labels input, single mask arrays
-
Method Summary
All Methods Static Methods Instance Methods Concrete Methods Modifier and Type Method Description List<MultiDataSet>
asList()
SplitV the MultiDataSet into a list of individual examples.MultiDataSet
copy()
Clone the datasetvoid
detach()
This method migrates this DataSet into current Workspace (if any)boolean
equals(Object o)
List<Serializable>
getExampleMetaData()
Get the example metadata, or null if no metadata has been set<T extends Serializable>
List<T>getExampleMetaData(Class<T> metaDataType)
Get the example metadata, or null if no metadata has been set
Note: this method results in an unchecked cast - care should be taken when using this!INDArray[]
getFeatures()
Get all of the input features, as an array of INDArraysINDArray
getFeatures(int index)
Get a single feature/input arrayINDArray
getFeaturesMaskArray(int index)
Get the specified feature mask array.INDArray[]
getFeaturesMaskArrays()
Get the feature mask arrays.INDArray[]
getLabels()
Get all of the labels, as an array of INDArraysINDArray
getLabels(int index)
Get a single label/output arrayINDArray
getLabelsMaskArray(int index)
Get the specified label mask array.INDArray[]
getLabelsMaskArrays()
Get the labels mask arrays.long
getMemoryFootprint()
This method returns memory used by this DataSetint
hashCode()
boolean
hasMaskArrays()
Whether there are any mask arrays (features or labels) present for this MultiDataSetboolean
isEmpty()
void
load(File from)
Load the contents of this MultiDataSet from the specified filevoid
load(InputStream from)
Load the contents of this MultiDataSet from the specified stream.static MultiDataSet
merge(Collection<? extends MultiDataSet> toMerge)
Merge a collection of MultiDataSet objects into a single MultiDataSet.void
migrate()
This method migrates this MultiDataSet into current Workspace (if any)int
numFeatureArrays()
Number of arrays of features/input data in the MultiDataSetint
numLabelsArrays()
Number of arrays of label/output data in the MultiDataSetvoid
save(File to)
Save this MultiDataSet to the specified filevoid
save(OutputStream to)
Save this MultiDataSet to the specified stream.void
setExampleMetaData(List<? extends Serializable> exampleMetaData)
Set the metadata for this MultiDataSet
By convention: the metadata can be any serializable object, one per example in the MultiDataSetvoid
setFeatures(int idx, INDArray features)
Set a single features array (by index) for the MultiDataSetvoid
setFeatures(INDArray[] features)
Set all of the features arrays for the MultiDataSetvoid
setFeaturesMaskArray(int idx, INDArray maskArray)
Set a single feature mask array by indexvoid
setFeaturesMaskArrays(INDArray[] maskArrays)
Set the feature mask arraysvoid
setLabels(int idx, INDArray labels)
Set a single labels array (by index) for the MultiDataSetvoid
setLabels(INDArray[] labels)
Set all of the labels arrays for the MultiDataSetvoid
setLabelsMaskArray(int idx, INDArray labelsMaskArray)
Set a single labels mask array by indexvoid
setLabelsMaskArray(INDArray[] labelsMaskArrays)
Set the labels mask arraysvoid
shuffle()
Shuffle the order of the examples in the MultiDataSet.String
toString()
-
-
-
Constructor Detail
-
MultiDataSet
public MultiDataSet()
Create a new (empty) MultiDataSet object (all fields are null)
-
MultiDataSet
public MultiDataSet(INDArray features, INDArray labels)
MultiDataSet constructor with single features/labels input, no mask arrays
-
MultiDataSet
public MultiDataSet(INDArray features, INDArray labels, INDArray featuresMask, INDArray labelsMask)
MultiDataSet constructor with single features/labels input, single mask arrays
-
MultiDataSet
public MultiDataSet(INDArray[] features, INDArray[] labels)
MultiDataSet constructor with no mask arrays
-
MultiDataSet
public MultiDataSet(INDArray[] features, INDArray[] labels, INDArray[] featuresMaskArrays, INDArray[] labelsMaskArrays)
- Parameters:
features
- The features (inputs) to the algorithm/neural networklabels
- The labels (outputs) to the algorithm/neural networkfeaturesMaskArrays
- The mask arrays for the features. May be null. Typically used with variable-length time series models, etclabelsMaskArrays
- The mask arrays for the labels. May be null. Typically used with variable-length time series models, etc
-
MultiDataSet
public MultiDataSet(INDArray[] features, INDArray[] labels, INDArray[] featuresMaskArrays, INDArray[] labelsMaskArrays, List<Serializable> exampleMetaData)
- Parameters:
features
- The features (inputs) to the algorithm/neural networklabels
- The labels (outputs) to the algorithm/neural networkfeaturesMaskArrays
- The mask arrays for the features. May be null. Typically used with variable-length time series models, etclabelsMaskArrays
- The mask arrays for the labels. May be null. Typically used with variable-length time series models, etcexampleMetaData
- Metadata for each example. May be null
-
-
Method Detail
-
getExampleMetaData
public List<Serializable> getExampleMetaData()
Description copied from interface:MultiDataSet
Get the example metadata, or null if no metadata has been set- Specified by:
getExampleMetaData
in interfaceMultiDataSet
- Returns:
- List of metadata instances
-
getExampleMetaData
public <T extends Serializable> List<T> getExampleMetaData(Class<T> metaDataType)
Description copied from interface:MultiDataSet
Get the example metadata, or null if no metadata has been set
Note: this method results in an unchecked cast - care should be taken when using this!- Specified by:
getExampleMetaData
in interfaceMultiDataSet
- Type Parameters:
T
- Type of metadata- Parameters:
metaDataType
- Class of the metadata (used for opType information)- Returns:
- List of metadata objects
-
setExampleMetaData
public void setExampleMetaData(List<? extends Serializable> exampleMetaData)
Description copied from interface:MultiDataSet
Set the metadata for this MultiDataSet
By convention: the metadata can be any serializable object, one per example in the MultiDataSet- Specified by:
setExampleMetaData
in interfaceMultiDataSet
- Parameters:
exampleMetaData
- Example metadata to set
-
numFeatureArrays
public int numFeatureArrays()
Description copied from interface:MultiDataSet
Number of arrays of features/input data in the MultiDataSet- Specified by:
numFeatureArrays
in interfaceMultiDataSet
-
numLabelsArrays
public int numLabelsArrays()
Description copied from interface:MultiDataSet
Number of arrays of label/output data in the MultiDataSet- Specified by:
numLabelsArrays
in interfaceMultiDataSet
-
getFeatures
public INDArray[] getFeatures()
Description copied from interface:MultiDataSet
Get all of the input features, as an array of INDArrays- Specified by:
getFeatures
in interfaceMultiDataSet
-
getFeatures
public INDArray getFeatures(int index)
Description copied from interface:MultiDataSet
Get a single feature/input array- Specified by:
getFeatures
in interfaceMultiDataSet
-
setFeatures
public void setFeatures(INDArray[] features)
Description copied from interface:MultiDataSet
Set all of the features arrays for the MultiDataSet- Specified by:
setFeatures
in interfaceMultiDataSet
-
setFeatures
public void setFeatures(int idx, INDArray features)
Description copied from interface:MultiDataSet
Set a single features array (by index) for the MultiDataSet- Specified by:
setFeatures
in interfaceMultiDataSet
-
getLabels
public INDArray[] getLabels()
Description copied from interface:MultiDataSet
Get all of the labels, as an array of INDArrays- Specified by:
getLabels
in interfaceMultiDataSet
-
getLabels
public INDArray getLabels(int index)
Description copied from interface:MultiDataSet
Get a single label/output array- Specified by:
getLabels
in interfaceMultiDataSet
-
setLabels
public void setLabels(INDArray[] labels)
Description copied from interface:MultiDataSet
Set all of the labels arrays for the MultiDataSet- Specified by:
setLabels
in interfaceMultiDataSet
-
setLabels
public void setLabels(int idx, INDArray labels)
Description copied from interface:MultiDataSet
Set a single labels array (by index) for the MultiDataSet- Specified by:
setLabels
in interfaceMultiDataSet
-
hasMaskArrays
public boolean hasMaskArrays()
Description copied from interface:MultiDataSet
Whether there are any mask arrays (features or labels) present for this MultiDataSet- Specified by:
hasMaskArrays
in interfaceMultiDataSet
-
getFeaturesMaskArrays
public INDArray[] getFeaturesMaskArrays()
Description copied from interface:MultiDataSet
Get the feature mask arrays. May return null if no feature mask arrays are present; otherwise, any entry in the array may be null if no mask array is present for that particular feature- Specified by:
getFeaturesMaskArrays
in interfaceMultiDataSet
-
getFeaturesMaskArray
public INDArray getFeaturesMaskArray(int index)
Description copied from interface:MultiDataSet
Get the specified feature mask array. Returns null if no feature mask is present for that particular feature/input- Specified by:
getFeaturesMaskArray
in interfaceMultiDataSet
-
setFeaturesMaskArrays
public void setFeaturesMaskArrays(INDArray[] maskArrays)
Description copied from interface:MultiDataSet
Set the feature mask arrays- Specified by:
setFeaturesMaskArrays
in interfaceMultiDataSet
-
setFeaturesMaskArray
public void setFeaturesMaskArray(int idx, INDArray maskArray)
Description copied from interface:MultiDataSet
Set a single feature mask array by index- Specified by:
setFeaturesMaskArray
in interfaceMultiDataSet
-
getLabelsMaskArrays
public INDArray[] getLabelsMaskArrays()
Description copied from interface:MultiDataSet
Get the labels mask arrays. May return null if no labels mask arrays are present; otherwise, any entry in the array may be null if no mask array is present for that particular label- Specified by:
getLabelsMaskArrays
in interfaceMultiDataSet
-
getLabelsMaskArray
public INDArray getLabelsMaskArray(int index)
Description copied from interface:MultiDataSet
Get the specified label mask array. Returns null if no label mask is present for that particular feature/input- Specified by:
getLabelsMaskArray
in interfaceMultiDataSet
-
setLabelsMaskArray
public void setLabelsMaskArray(INDArray[] labelsMaskArrays)
Description copied from interface:MultiDataSet
Set the labels mask arrays- Specified by:
setLabelsMaskArray
in interfaceMultiDataSet
-
setLabelsMaskArray
public void setLabelsMaskArray(int idx, INDArray labelsMaskArray)
Description copied from interface:MultiDataSet
Set a single labels mask array by index- Specified by:
setLabelsMaskArray
in interfaceMultiDataSet
-
save
public void save(OutputStream to) throws IOException
Description copied from interface:MultiDataSet
Save this MultiDataSet to the specified stream. Stream will be closed after saving.- Specified by:
save
in interfaceMultiDataSet
- Throws:
IOException
-
save
public void save(File to) throws IOException
Description copied from interface:MultiDataSet
Save this MultiDataSet to the specified file- Specified by:
save
in interfaceMultiDataSet
- Throws:
IOException
-
load
public void load(InputStream from) throws IOException
Description copied from interface:MultiDataSet
Load the contents of this MultiDataSet from the specified stream. Stream will be closed after loading.- Specified by:
load
in interfaceMultiDataSet
- Throws:
IOException
-
load
public void load(File from) throws IOException
Description copied from interface:MultiDataSet
Load the contents of this MultiDataSet from the specified file- Specified by:
load
in interfaceMultiDataSet
- Throws:
IOException
-
asList
public List<MultiDataSet> asList()
Description copied from interface:MultiDataSet
SplitV the MultiDataSet into a list of individual examples.- Specified by:
asList
in interfaceMultiDataSet
- Returns:
- List of MultiDataSets, each with 1 example
-
copy
public MultiDataSet copy()
Clone the dataset- Specified by:
copy
in interfaceMultiDataSet
- Returns:
- a clone of the dataset
-
merge
public static MultiDataSet merge(Collection<? extends MultiDataSet> toMerge)
Merge a collection of MultiDataSet objects into a single MultiDataSet. Merging is done by concatenating along dimension 0 (example number in batch) Merging operation may introduce mask arrays (when necessary) for time series data that has different lengths; if mask arrays already exist, these will be merged also.- Parameters:
toMerge
- Collection of MultiDataSet objects to merge- Returns:
- a single MultiDataSet object, containing the arrays of
-
getMemoryFootprint
public long getMemoryFootprint()
This method returns memory used by this DataSet- Specified by:
getMemoryFootprint
in interfaceMultiDataSet
- Returns:
-
migrate
public void migrate()
Description copied from interface:MultiDataSet
This method migrates this MultiDataSet into current Workspace (if any)- Specified by:
migrate
in interfaceMultiDataSet
-
detach
public void detach()
This method migrates this DataSet into current Workspace (if any)- Specified by:
detach
in interfaceMultiDataSet
-
isEmpty
public boolean isEmpty()
- Specified by:
isEmpty
in interfaceMultiDataSet
- Returns:
- True if the MultiDataSet is empty (all features/labels arrays are empty)
-
shuffle
public void shuffle()
Description copied from interface:MultiDataSet
Shuffle the order of the examples in the MultiDataSet. Note that this generally won't make any difference in practice unless the MultiDataSet is later split.- Specified by:
shuffle
in interfaceMultiDataSet
-
-