org.deeplearning4j.optimize.solvers.accumulation.encoding.threshold

Class TargetSparsityThresholdAlgorithm

java.lang.Object

org.deeplearning4j.optimize.solvers.accumulation.encoding.threshold.TargetSparsityThresholdAlgorithm

All Implemented Interfaces:

Serializable, ThresholdAlgorithm

public class TargetSparsityThresholdAlgorithm
extends Object
implements ThresholdAlgorithm

Targets a specific sparisty throughout training

Author:

Alex Black

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
static double	DEFAULT_DECAY_RATE
static double	DEFAULT_INITIAL_THRESHOLD
static double	DEFAULT_SPARSITY_TARGET

Constructor Summary

Constructors

Constructor and Description
TargetSparsityThresholdAlgorithm() Create the adaptive threshold algorithm with the default initial threshold DEFAULT_INITIAL_THRESHOLD, default sparsity target DEFAULT_SPARSITY_TARGET and default decay rate DEFAULT_DECAY_RATE
TargetSparsityThresholdAlgorithm(double initialThreshold, double sparsityTarget, double decayRate)

Method Summary

Modifier and Type	Method and Description
double	calculateThreshold(int iteration, int epoch, Double lastThreshold, Boolean lastWasDense, Double lastSparsityRatio, INDArray updatesPlusResidual)
TargetSparsityThresholdAlgorithm	clone()
ThresholdAlgorithmReducer	newReducer() Create a new ThresholdAlgorithmReducer.
String	toString()

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

DEFAULT_INITIAL_THRESHOLD

public static final double DEFAULT_INITIAL_THRESHOLD

See Also:

Constant Field Values

DEFAULT_SPARSITY_TARGET

public static final double DEFAULT_SPARSITY_TARGET

See Also:

Constant Field Values

DEFAULT_DECAY_RATE

public static final double DEFAULT_DECAY_RATE

Constructor Detail

TargetSparsityThresholdAlgorithm

public TargetSparsityThresholdAlgorithm()

Create the adaptive threshold algorithm with the default initial threshold DEFAULT_INITIAL_THRESHOLD, default sparsity target DEFAULT_SPARSITY_TARGET and default decay rate DEFAULT_DECAY_RATE

TargetSparsityThresholdAlgorithm

public TargetSparsityThresholdAlgorithm(double initialThreshold,
                                        double sparsityTarget,
                                        double decayRate)

Parameters:

initialThreshold - The initial threshold to use

sparsityTarget - The sparsity target

decayRate - The decay rate. For example 0.95

Method Detail

calculateThreshold

public double calculateThreshold(int iteration,
                                 int epoch,
                                 Double lastThreshold,
                                 Boolean lastWasDense,
                                 Double lastSparsityRatio,
                                 INDArray updatesPlusResidual)

Specified by:

calculateThreshold in interface ThresholdAlgorithm

Parameters:

iteration - Current neural network training iteration

epoch - Current neural network training epoch

lastThreshold - The encoding threshold used in the last iteration - if available. May be null for first iteration in an epoch (where no 'last iteration' value is available)

lastWasDense - Whether the last encoding was dense (true) or sparse (false). May be null for the first iteration in an epoch (where no 'last iteration' value is available)

lastSparsityRatio - The sparsity ratio of the last iteration. Sparsity ratio is defined as numElements(encoded)/length(updates). A sparsity ratio of 1.0 would mean all entries present in encoded representation; a sparsity ratio of 0.0 would mean the encoded vector did not contain any values. Note: when the last encoding was dense, lastSparsityRatio is always null - this means that the sparsity ratio is larger than 1/16 = 0.0625

updatesPlusResidual - The actual array (updates plus residual) that will be encoded using the threshold calculated/returned by this method

Returns:

newReducer

public ThresholdAlgorithmReducer newReducer()

Description copied from interface: ThresholdAlgorithm

Create a new ThresholdAlgorithmReducer. Note that implementations should NOT add the curret ThresholdAlgorithm to it.

Specified by:

newReducer in interface ThresholdAlgorithm

Returns:

ThresholdAlgorithmReducer

clone

public TargetSparsityThresholdAlgorithm clone()

Specified by:

clone in interface ThresholdAlgorithm

Overrides:

clone in class Object

Returns:

A clone of the current threshold algorithm

toString

public String toString()

Overrides:

toString in class Object