Package ai.djl.training.optimizer

Class Adam

java.lang.Object

ai.djl.training.optimizer.Optimizer

ai.djl.training.optimizer.Adam

public class Adam
extends Optimizer

Adam is a generalization of the AdaGrad Optimizer.
\( grad += weight_decay * w\)
\( m = beta1 * m + (1 - beta1) * grad\)
\( v = beta2 * v + (1 - beta2) * grad^2 \)
\( learning_rate_bias_correction = learning_rate / beta1**t * sqrt(beta2**t) \)
\( w -= learning_rate_bias_correction * m / (sqrt(v) + epsilon) \)

where g represents the gradient, and m/v are 1st and 2nd order moment estimates (mean and variance), t is the step.

See Also:

The D2L chapter on Adam

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	Adam.Builder	The Builder to construct an Adam object.

Nested classes/interfaces inherited from class ai.djl.training.optimizer.Optimizer

Optimizer.OptimizerBuilder<T extends Optimizer.OptimizerBuilder>

Field Summary

Fields inherited from class ai.djl.training.optimizer.Optimizer

clipGrad, rescaleGrad

Constructor Summary

Constructors

Modifier	Constructor	Description
protected	Adam(Adam.Builder builder)	Creates a new instance of Adam optimizer.

Method Summary

Modifier and Type	Method	Description
static Adam.Builder	builder()	Creates a builder to build a Adam.
void	update(java.lang.String parameterId, NDArray weight, NDArray grad)	Updates the parameters according to the gradients.

Methods inherited from class ai.djl.training.optimizer.Optimizer

adadelta, adagrad, adam, adamW, getWeightDecay, nag, rmsprop, sgd, updateCount, withDefaultState

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Adam

protected Adam(Adam.Builder builder)

Creates a new instance of Adam optimizer.

Parameters:

builder - the builder to create a new instance of Adam optimizer

Method Detail

update

public void update(java.lang.String parameterId,
                   NDArray weight,
                   NDArray grad)

Updates the parameters according to the gradients.

Specified by:

update in class Optimizer

Parameters:

parameterId - the parameter to be updated

weight - the weights of the parameter

grad - the gradients

builder

public static Adam.Builder builder()

Creates a builder to build a Adam.

Returns:

a new builder