public class XavierInitializer extends java.lang.Object implements Initializer
XavierInitializer is an Initializer that performs "Xavier" initialization for
parameters. This initializer is designed to keep the scale of gradients roughly the same in all
layers. It was originally defined in the paper Understanding the difficulty of
training deep feedforward neural networks.
XavierInitializer is specified by the type of random distribution(XavierInitializer.RandomType),
the factor type(XavierInitializer.FactorType), and the magnitude of the scale. By default, XavierInitializer.RandomType is UNIFORM and XavierInitializer.FactorType is AVG. The initializer fills the
weights with random numbers in the range of \([-c, c]\), where \(c = \sqrt{\frac{3.}{0.5 *
(n_{in} + n_{out})}}\) where \(n_{in}\) is the number of neurons feeding into weights, and
\(n_{out}\) is the number of neurons the result is fed to.
If XavierInitializer.RandomType is UNIFORM and XavierInitializer.FactorType is IN, then \(c =
\sqrt{\frac{3.}{n_{in}}}\). Similarly when XavierInitializer.FactorType is OUT, then \(c =
\sqrt{\frac{3.}{n_{out}}}\).
If XavierInitializer.RandomType is GAUSSIAN and XavierInitializer.FactorType is AVG, the
initializer fills the weights with numbers from normal distribution with a standard deviation of
\(\sqrt{\frac{3.}{0.5 * (n_{in} + n_{out})}}\).
Another common setting of the XavierInitializer is defined in the paper Delving Deep into Rectifiers: Surpassing Human-Level
Performance on ImageNet Classification. These settings better handle non-linearity when
preserving the variance across layers in a neural network. It can be initialized with new
XavierInitializer(RandomType.GAUSSIAN, FactorType.IN, 2)).
| Modifier and Type | Class and Description |
|---|---|
static class |
XavierInitializer.FactorType
Enum for different types of factor type.
|
static class |
XavierInitializer.RandomType
Enum for different types of random distributions.
|
ONES, ZEROS| Constructor and Description |
|---|
XavierInitializer()
Creates a new instance of
XavierInitializer. |
XavierInitializer(XavierInitializer.RandomType randomType,
XavierInitializer.FactorType factorType,
float magnitude)
Initializes a Xavier initializer.
|
| Modifier and Type | Method and Description |
|---|---|
NDArray |
initialize(NDManager manager,
Shape shape,
DataType dataType)
Initializes a single
NDArray. |
public XavierInitializer(XavierInitializer.RandomType randomType, XavierInitializer.FactorType factorType, float magnitude)
randomType - the random generator type, can be GAUSSIAN or UNIFORMfactorType - the factor type, can be one of AVG, IN, or OUTmagnitude - the scale of the random numberpublic XavierInitializer()
XavierInitializer.public NDArray initialize(NDManager manager, Shape shape, DataType dataType)
NDArray.initialize in interface Initializermanager - the NDManager to create the new NDArray inshape - the Shape for the new NDArraydataType - the DataType for the new NDArrayNDArray initialized with the manager and shape