org.deeplearning4j.nn.conf.dropout

Class Dropout

java.lang.Object

org.deeplearning4j.nn.conf.dropout.Dropout

All Implemented Interfaces:

Serializable, Cloneable, IDropout

public class Dropout
extends Object
implements IDropout

Implements standard (inverted) dropout.

Regarding dropout probability. This is the probability of retaining each input activation value for a layer. Thus, each input activation x is independently set to:
x <- 0, with probability 1-p
x <- x/p with probability p
Note that this "inverted" dropout scheme maintains the expected value of activations - i.e., E(x) is the same before and after dropout.
Dropout schedules (i.e., varying probability p as a function of iteration/epoch) are also supported.

Other libraries (notably, Keras) use p == probability(dropping an activation)
In DL4J, new Dropout(x) will keep an input activation with probability x, and set to 0 with probability 1-x.
Thus, a dropout value of 1.0 is functionally equivalent to no dropout: i.e., 100% probability of retaining each input activation.

Note 1: As per all IDropout instances, dropout is applied at training time only - and is automatically not applied at test time (for evaluation, etc)
Note 2: Care should be taken when setting lower (probability of retaining) values for (too much information may be lost with aggressive (very low) dropout values).
Note 3: Frequently, dropout is not applied to (or, has higher retain probability for) input (first layer) layers. Dropout is also often not applied to output layers.
Note 4: Implementation detail (most users can ignore): DL4J uses inverted dropout, as described here: http://cs231n.github.io/neural-networks-2/

See: Srivastava et al. 2014: Dropout: A Simple Way to Prevent Neural Networks from Overfitting http://www.cs.toronto.edu/~rsalakhu/papers/srivastava14a.pdf

Author:

Alex Black

See Also:

Serialized Form

Constructor Summary

Constructors

Modifier	Constructor and Description
	Dropout(double activationRetainProbability)
protected	Dropout(double activationRetainProbability, org.nd4j.linalg.schedule.ISchedule activationRetainProbabilitySchedule)
	Dropout(org.nd4j.linalg.schedule.ISchedule activationRetainProbabilitySchedule)

Method Summary

Modifier and Type	Method and Description
org.nd4j.linalg.api.ndarray.INDArray	applyDropout(org.nd4j.linalg.api.ndarray.INDArray inputActivations, int iteration, int epoch, boolean inPlace)
Dropout	clone()

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Dropout

public Dropout(double activationRetainProbability)

Parameters:

activationRetainProbability - Probability of retaining an activation - see Dropout javadoc

Dropout

public Dropout(org.nd4j.linalg.schedule.ISchedule activationRetainProbabilitySchedule)

Parameters:

activationRetainProbabilitySchedule - Schedule for probability of retaining an activation - see Dropout javadoc

Dropout

protected Dropout(double activationRetainProbability,
                  org.nd4j.linalg.schedule.ISchedule activationRetainProbabilitySchedule)

Method Detail

applyDropout

public org.nd4j.linalg.api.ndarray.INDArray applyDropout(org.nd4j.linalg.api.ndarray.INDArray inputActivations,
                                                         int iteration,
                                                         int epoch,
                                                         boolean inPlace)

Specified by:

applyDropout in interface IDropout

Parameters:

inputActivations - Input activations array

iteration - Current iteration number

epoch - Current epoch number

inPlace - If true: modify the input activations in-place. False: Copy the input activations and apply dropout on the copy instead

Returns:

clone

public Dropout clone()

Specified by:

clone in interface IDropout

Overrides:

clone in class Object