org.deeplearning4j.nn.conf.dropout

Class AlphaDropout

java.lang.Object

org.deeplearning4j.nn.conf.dropout.AlphaDropout

All Implemented Interfaces:

Serializable, Cloneable, IDropout

public class AlphaDropout
extends Object
implements IDropout

AlphaDropout is a dropout technique proposed by Klaumbauer et al. 2017 - Self-Normalizing Neural Networks https://arxiv.org/abs/1706.02515

This dropout technique was designed specifically for self-normalizing neural networks - i.e., networks using ActivationSELU / Activation.SELU activation function, combined with the N(0,stdev=1/sqrt(fanIn)) "SNN" weight initialization, WeightInit.NORMAL

In conjuction with the aforementioned activation function and weight initialization, AlphaDropout attempts to keep both the mean and variance of the post-dropout activations to the same (in expectation) as before alpha dropout was applied.
Specifically, AlphaDropout implements a * (x * d + alphaPrime * (1-d)) + b, where d ~ Bernoulli(p), i.e., d \in {0,1}. Where x is the input activations, a, b, alphaPrime are constants determined from the SELU alpha/lambda parameters. Users should use the default alpha/lambda values in virtually all cases.

Dropout schedules (i.e., varying probability p as a function of iteration/epoch) are also supported.

Author:

Alex Black

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
static double	DEFAULT_ALPHA
static double	DEFAULT_LAMBDA

Constructor Summary

Constructors

Modifier	Constructor and Description
	AlphaDropout(double activationRetainProbability)
protected	AlphaDropout(double activationRetainProbability, ISchedule activationRetainProbabilitySchedule, double alpha, double lambda)
	AlphaDropout(@NonNull ISchedule activationRetainProbabilitySchedule)

Method Summary

Modifier and Type	Method and Description
double	a(double p)
INDArray	applyDropout(INDArray inputActivations, INDArray output, int iteration, int epoch, LayerWorkspaceMgr workspaceMgr)
double	b(double p)
INDArray	backprop(INDArray gradAtOutput, INDArray gradAtInput, int iteration, int epoch) Perform backprop.
void	clear() Clear the internal state (for example, dropout mask) if any is present
AlphaDropout	clone()

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DEFAULT_ALPHA

public static final double DEFAULT_ALPHA

See Also:

Constant Field Values

DEFAULT_LAMBDA

public static final double DEFAULT_LAMBDA

See Also:

Constant Field Values

Constructor Detail

AlphaDropout

public AlphaDropout(double activationRetainProbability)

Parameters:

activationRetainProbability - Probability of retaining an activation. See AlphaDropout javadoc

AlphaDropout

public AlphaDropout(@NonNull
                    @NonNull ISchedule activationRetainProbabilitySchedule)

Parameters:

activationRetainProbabilitySchedule - Schedule for the probability of retaining an activation. See AlphaDropout javadoc

AlphaDropout

protected AlphaDropout(double activationRetainProbability,
                       ISchedule activationRetainProbabilitySchedule,
                       double alpha,
                       double lambda)

Method Detail

applyDropout

public INDArray applyDropout(INDArray inputActivations,
                             INDArray output,
                             int iteration,
                             int epoch,
                             LayerWorkspaceMgr workspaceMgr)

Specified by:

applyDropout in interface IDropout

Parameters:

inputActivations - Input activations array

output - The result array (same as inputArray for in-place ops) for the post-dropout activations

iteration - Current iteration number

epoch - Current epoch number

workspaceMgr - Workspace manager, if any storage is required (use ArrayType.INPUT)

Returns:

The output (resultArray) after applying dropout

backprop

public INDArray backprop(INDArray gradAtOutput,
                         INDArray gradAtInput,
                         int iteration,
                         int epoch)

Description copied from interface: IDropout

Perform backprop. This should also clear the internal state (dropout mask) if any is present

Specified by:

backprop in interface IDropout

Parameters:

gradAtOutput - Gradients at the output of the dropout op - i.e., dL/dOut

gradAtInput - Gradients at the input of the dropout op - i.e., dL/dIn. Use the same array as gradAtOutput to apply the backprop gradient in-place

iteration - Current iteration

epoch - Current epoch

Returns:

Same array as gradAtInput - i.e., gradient after backpropagating through dropout op - i.e., dL/dIn

clear

public void clear()

Description copied from interface: IDropout

Clear the internal state (for example, dropout mask) if any is present

Specified by:

clear in interface IDropout

clone

public AlphaDropout clone()

Specified by:

clone in interface IDropout

Overrides:

clone in class Object

a

public double a(double p)

b

public double b(double p)