org.deeplearning4j.nn.layers

Class BasePretrainNetwork<LayerConfT extends BasePretrainNetwork>

java.lang.Object

org.deeplearning4j.nn.layers.AbstractLayer<LayerConfT>

org.deeplearning4j.nn.layers.BaseLayer<LayerConfT>

org.deeplearning4j.nn.layers.BasePretrainNetwork<LayerConfT>

All Implemented Interfaces:

Serializable, Cloneable, Layer, Model, Trainable

Direct Known Subclasses:

AutoEncoder

public abstract class BasePretrainNetwork<LayerConfT extends BasePretrainNetwork>
extends BaseLayer<LayerConfT>

Baseline class for any Neural Network used as a layer in a deep network *

Author:

Adam Gibson

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from interface org.deeplearning4j.nn.api.Layer

Layer.TrainingMode, Layer.Type

Field Summary

Fields inherited from class org.deeplearning4j.nn.layers.BaseLayer

gradient, gradientsFlattened, gradientViews, optimizer, params, paramsFlattened, score, solver, weightNoiseParams

Fields inherited from class org.deeplearning4j.nn.layers.AbstractLayer

cacheMode, conf, dataType, dropoutApplied, epochCount, index, input, inputModificationAllowed, iterationCount, maskArray, maskState, preOutput, trainingListeners

Constructor Summary

Constructors

Constructor and Description
BasePretrainNetwork(NeuralNetConfiguration conf, DataType dataType)

Method Summary

Modifier and Type	Method and Description
Pair<Gradient,INDArray>	backpropGradient(INDArray epsilon, LayerWorkspaceMgr workspaceMgr) Calculate the gradient relative to the error in the next layer
double	calcRegularizationScore(boolean backpropParamsOnly) Calculate the regularization component of the score, for the parameters in this layer For example, the L1, L2 and/or weight decay components of the loss function
protected Gradient	createGradient(INDArray wGradient, INDArray vBiasGradient, INDArray hBiasGradient)
INDArray	getCorruptedInput(INDArray x, double corruptionLevel) Corrupts the given input by doing a binomial sampling given the corruption level
long	numParams() The number of parameters for the model, for backprop (i.e., excluding visible bias)
long	numParams(boolean backwards) the number of parameters for the model
INDArray	params() Returns the parameters of the neural network as a flattened row vector
Map<String,INDArray>	paramTable(boolean backpropParamsOnly) Table of parameters by key, for backprop For many models (dense layers, etc) - all parameters are backprop parameters
abstract Pair<INDArray,INDArray>	sampleHiddenGivenVisible(INDArray v) Sample the hidden distribution given the visible
abstract Pair<INDArray,INDArray>	sampleVisibleGivenHidden(INDArray h) Sample the visible distribution given the hidden
void	setParams(INDArray params) Set the parameters for this model.
protected void	setScoreWithZ(INDArray z)

Methods inherited from class org.deeplearning4j.nn.layers.BaseLayer

activate, clear, clearNoiseWeightParams, clone, computeGradientAndScore, fit, fit, getGradientsViewArray, getOptimizer, getParam, getParamWithNoise, gradient, hasBias, hasLayerNorm, layerConf, paramTable, preOutput, preOutputWithPreNorm, score, setBackpropGradientsViewArray, setParam, setParams, setParamsViewArray, setParamTable, toString, update, update

Methods inherited from class org.deeplearning4j.nn.layers.AbstractLayer

activate, addListeners, allowInputModification, applyConstraints, applyDropOutIfNecessary, applyMask, assertInputSet, backpropDropOutIfPresent, batchSize, close, conf, feedForwardMaskArray, getConfig, getEpochCount, getHelper, getIndex, getInput, getInputMiniBatchSize, getListeners, getMaskArray, gradientAndScore, init, input, layerId, setCacheMode, setConf, setEpochCount, setIndex, setInput, setInputMiniBatchSize, setListeners, setListeners, setMaskArray, type, updaterDivideByMinibatch

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.deeplearning4j.nn.api.Layer

getIterationCount, isPretrainLayer, setIterationCount

Constructor Detail

BasePretrainNetwork

public BasePretrainNetwork(NeuralNetConfiguration conf,
                           DataType dataType)

Method Detail

getCorruptedInput

public INDArray getCorruptedInput(INDArray x,
                                  double corruptionLevel)

Corrupts the given input by doing a binomial sampling given the corruption level

Parameters:

x - the input to corrupt

corruptionLevel - the corruption value

Returns:

the binomial sampled corrupted input

createGradient

protected Gradient createGradient(INDArray wGradient,
                                  INDArray vBiasGradient,
                                  INDArray hBiasGradient)

numParams

public long numParams(boolean backwards)

Description copied from interface: Model

the number of parameters for the model

Specified by:

numParams in interface Model

Overrides:

numParams in class AbstractLayer<LayerConfT extends BasePretrainNetwork>

Returns:

the number of parameters for the model

sampleHiddenGivenVisible

public abstract Pair<INDArray,INDArray> sampleHiddenGivenVisible(INDArray v)

Sample the hidden distribution given the visible

Parameters:

v - the visible to sample from

Returns:

the hidden mean and sample

sampleVisibleGivenHidden

public abstract Pair<INDArray,INDArray> sampleVisibleGivenHidden(INDArray h)

Sample the visible distribution given the hidden

Parameters:

h - the hidden to sample from

Returns:

the mean and sample

setScoreWithZ

protected void setScoreWithZ(INDArray z)

Overrides:

setScoreWithZ in class BaseLayer<LayerConfT extends BasePretrainNetwork>

paramTable

public Map<String,INDArray> paramTable(boolean backpropParamsOnly)

Description copied from interface: Model

Table of parameters by key, for backprop For many models (dense layers, etc) - all parameters are backprop parameters

Specified by:

paramTable in interface Model

Specified by:

paramTable in interface Trainable

Overrides:

paramTable in class BaseLayer<LayerConfT extends BasePretrainNetwork>

Parameters:

backpropParamsOnly - If true, return backprop params only. If false: return all params (equivalent to paramsTable())

Returns:

Parameter table

params

public INDArray params()

Description copied from class: BaseLayer

Returns the parameters of the neural network as a flattened row vector

Specified by:

params in interface Model

Specified by:

params in interface Trainable

Overrides:

params in class BaseLayer<LayerConfT extends BasePretrainNetwork>

Returns:

the parameters of the neural network

numParams

public long numParams()

The number of parameters for the model, for backprop (i.e., excluding visible bias)

Specified by:

numParams in interface Model

Specified by:

numParams in interface Trainable

Overrides:

numParams in class BaseLayer<LayerConfT extends BasePretrainNetwork>

Returns:

the number of parameters for the model (ex. visible bias)

setParams

public void setParams(INDArray params)

Description copied from interface: Model

Set the parameters for this model. This expects a linear ndarray which then be unpacked internally relative to the expected ordering of the model

Specified by:

setParams in interface Model

Overrides:

setParams in class BaseLayer<LayerConfT extends BasePretrainNetwork>

Parameters:

params - the parameters for the model

backpropGradient

public Pair<Gradient,INDArray> backpropGradient(INDArray epsilon,
                                                LayerWorkspaceMgr workspaceMgr)

Description copied from interface: Layer

Calculate the gradient relative to the error in the next layer

Specified by:

backpropGradient in interface Layer

Overrides:

backpropGradient in class BaseLayer<LayerConfT extends BasePretrainNetwork>

Parameters:

epsilon - w^(L+1)*delta^(L+1). Or, equiv: dC/da, i.e., (dC/dz)*(dz/da) = dC/da, where C is cost function a=sigma(z) is activation.

workspaceMgr - Workspace manager

Returns:

Pair where Gradient is gradient for this layer, INDArray is epsilon (activation gradient) needed by next layer, but before element-wise multiply by sigmaPrime(z). So for standard feed-forward layer, if this layer is L, then return.getSecond() == dL/dIn = (w^(L)*(delta^(L))^T)^T. Note that the returned array should be placed in the ArrayType.ACTIVATION_GRAD workspace via the workspace manager

calcRegularizationScore

public double calcRegularizationScore(boolean backpropParamsOnly)

Description copied from interface: Layer

Calculate the regularization component of the score, for the parameters in this layer
For example, the L1, L2 and/or weight decay components of the loss function

Specified by:

calcRegularizationScore in interface Layer

Overrides:

calcRegularizationScore in class BaseLayer<LayerConfT extends BasePretrainNetwork>

Parameters:

backpropParamsOnly - If true: calculate regularization score based on backprop params only. If false: calculate based on all params (including pretrain params, if any)

Returns:

the regularization score of