ai.djl.nn.norm

Class BatchNorm

java.lang.Object

ai.djl.nn.AbstractBlock

ai.djl.nn.ParameterBlock

ai.djl.nn.norm.BatchNorm

All Implemented Interfaces:

Block

public class BatchNorm
extends ParameterBlock

In batch training (training with more than one samples per iteration), a batch normalization layer works by normalizing the values of input data to have mean of 0 and variance of 1. Since this may alter the representation of a layer, two parameters (\ (\gamma\) and \(\beta\)) are learned along the normalization process to respectively scale and shift the normalized output (activations) to have any mean and variance so the network can utilize non-linear transformations such as sigmoid function as described in the paper. During backpropagation, both \(\gamma\) and \(\beta\) parameters are included following the chain-rule in derivation.

The problem of varying distribution of input data requires the training process of a deep network to compensate for each different data distribution per batch, hence changing parameters' values as new batch data is processed and changes distribution of the network's (and each of its layers) activations. This condition is termed as internal covariate shift, and such occurrence prevents the network to learn faster and generalize better to unseen data.

With batch normalization, one benefits by having faster learning process as batch normalization allows larger learning rate without causing gradient problems on backpropagation as all inputs are normalized and hence reducing the scale of weight update impact on backpropagation. In some cases, the utilization of batch normalization layer regularizes the network and reduces, even eliminates, the need for dropout, which in turn results in even faster training process since dropout slows down training by 2-3 times. However, it was reported that batch normalization may not be beneficial when small batch sizes are used.

Formally, batch normalization is represented below:
\(\hat{x} \:=\: \frac{x \:-\: \mu_{batch}}{\sqrt{\sigma^2_{batch} \:+\: \epsilon}}\),
where \(\hat{x}\) is the normalized input, \(\mu_{batch}\) and \(\sigma^2_{batch}\) respectively denote the mean and variance of a batch, and \(\epsilon\) (epsilon) is a constant for numerical stability. The scale and shift operation can be formally defined as follows:
\(y \:=\: \gamma\hat{x} \:+\: \beta\),
where \(\gamma\) is the scale factor and \(\beta\) is the shift factor.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	BatchNorm.Builder The Builder to construct a BatchNorm.

Field Summary

Fields inherited from class ai.djl.nn.AbstractBlock

inputNames, inputShapes

Method Summary

Modifier and Type	Method and Description
void	beforeInitialize(Shape[] inputShapes) Performs any action necessary before initialization.
static BatchNorm.Builder	builder() Creates a builder to build a BatchNorm.
NDList	forward(ParameterStore parameterStore, NDList inputs, boolean training, ai.djl.util.PairList<java.lang.String,java.lang.Object> params) Applies the operating function of the block once.
java.util.List<Parameter>	getDirectParameters() Returns a list of all the direct parameters of the block.
Shape[]	getOutputShapes(NDManager manager, Shape[] inputShapes) Returns the expected output shapes of the block for the specified input shapes.
Shape	getParameterShape(java.lang.String name, Shape[] inputShapes) Returns the shape of the specified direct parameter of this block given the shapes of the input to the block.
void	loadParameters(NDManager manager, java.io.DataInputStream is) Loads the parameters from the given input stream.
void	saveParameters(java.io.DataOutputStream os) Writes the parameters of the block to the given outputStream.

Methods inherited from class ai.djl.nn.ParameterBlock

getChildren, initialize, toString

Methods inherited from class ai.djl.nn.AbstractBlock

cast, clear, describeInput, getParameters, isInitialized, readInputShapes, saveInputShapes, setInitializer, setInitializer

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface ai.djl.nn.Block

forward, validateLayout

Method Detail

forward

public NDList forward(ParameterStore parameterStore,
                      NDList inputs,
                      boolean training,
                      ai.djl.util.PairList<java.lang.String,java.lang.Object> params)

Applies the operating function of the block once. This method should be called only on blocks that are initialized.

Parameters:

parameterStore - the parameter store

inputs - the input NDList

training - true for a training forward pass

params - optional parameters

Returns:

the output of the forward pass

getOutputShapes

public Shape[] getOutputShapes(NDManager manager,
                               Shape[] inputShapes)

Returns the expected output shapes of the block for the specified input shapes.

Parameters:

manager - an NDManager

inputShapes - the shapes of the inputs

Returns:

the expected output shapes of the block

getDirectParameters

public java.util.List<Parameter> getDirectParameters()

Returns a list of all the direct parameters of the block.

Returns:

the list of Parameter

beforeInitialize

public void beforeInitialize(Shape[] inputShapes)

Performs any action necessary before initialization.

Overrides:

beforeInitialize in class AbstractBlock

Parameters:

inputShapes - the expected shapes of the input

getParameterShape

public Shape getParameterShape(java.lang.String name,
                               Shape[] inputShapes)

Returns the shape of the specified direct parameter of this block given the shapes of the input to the block.

Parameters:

name - the name of the parameter

inputShapes - the shapes of the input to the block

Returns:

the shape of the parameter specified

saveParameters

public void saveParameters(java.io.DataOutputStream os)
                    throws java.io.IOException

Writes the parameters of the block to the given outputStream.

Parameters:

os - the outputstream to save the parameters to

Throws:

java.io.IOException - if an I/O error occurs

loadParameters

public void loadParameters(NDManager manager,
                           java.io.DataInputStream is)
                    throws java.io.IOException,
                           MalformedModelException

Loads the parameters from the given input stream.

Parameters:

manager - an NDManager to create the parameter arrays

is - the inputstream that stream the parameter values

Throws:

java.io.IOException - if an I/O error occurs

MalformedModelException - if the model file is corrupted or unsupported

builder

public static BatchNorm.Builder builder()

Creates a builder to build a BatchNorm.

Returns:

a new builder