org.nd4j.linalg.learning

Class AdaGrad

java.lang.Object

org.nd4j.linalg.learning.AdaGrad

All Implemented Interfaces:

Serializable, GradientUpdater

public class AdaGrad
extends Object
implements Serializable, GradientUpdater

Vectorized Learning Rate used per Connection Weight

Adapted from: http://xcorr.net/2014/01/23/adagrad-eliminating-learning-rates-in-stochastic-gradient-descent/ See also http://cs231n.github.io/neural-networks-3/#ada

Author:

Adam Gibson

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	AdaGrad.AdaGradAggregator

Field Summary

Fields

Modifier and Type	Field and Description
INDArray	historicalGradient
protected double	learningRate
protected int	numIterations
int[]	shape

Constructor Summary

Constructors

Constructor and Description
AdaGrad(double learningRate)
AdaGrad(double learningRate, double epsilon)
AdaGrad(int[] shape, double learningRate)
AdaGrad(int rows, int cols)
AdaGrad(int rows, int cols, double learningRate)

Method Summary

Modifier and Type	Method and Description
AdaGrad	createSubset(int index)
GradientUpdaterAggregator	getAggregator(boolean addThis) Get a GradientUpdaterAggregator.
double	getGradient(double gradient, int column, int[] shape)
INDArray	getGradient(INDArray gradient, int iteration) Gets feature specific learning rates Adagrad keeps a history of gradients being passed in.
INDArray	getGradient(INDArray gradient, int slice, int[] shape)
void	setStateViewArray(INDArray viewArray, int[] gradientShape, char gradientOrder, boolean initialize) For the internal updater state (if any): set this to use the provided array.
int	stateSizeForInputSize(int inputSize) For a give input size (length) array, how big is the internal state? Typically 0, 1 or 2x the input size, depending on the type of updater
void	update(Object... args) update(learningRate,momentum)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

historicalGradient

public INDArray historicalGradient

shape

public int[] shape

learningRate

protected double learningRate

numIterations

protected int numIterations

Constructor Detail

AdaGrad

public AdaGrad(int rows,
               int cols,
               double learningRate)

Parameters:

rows -

cols -

learningRate -

AdaGrad

public AdaGrad(int rows,
               int cols)

AdaGrad

public AdaGrad(int[] shape,
               double learningRate)

AdaGrad

public AdaGrad(double learningRate)

AdaGrad

public AdaGrad(double learningRate,
               double epsilon)

Method Detail

stateSizeForInputSize

public int stateSizeForInputSize(int inputSize)

Description copied from interface: GradientUpdater

For a give input size (length) array, how big is the internal state? Typically 0, 1 or 2x the input size, depending on the type of updater

Specified by:

stateSizeForInputSize in interface GradientUpdater

Parameters:

inputSize - Length of the input array

Returns:

Number of elements in the internal state

setStateViewArray

public void setStateViewArray(INDArray viewArray,
                              int[] gradientShape,
                              char gradientOrder,
                              boolean initialize)

Description copied from interface: GradientUpdater

For the internal updater state (if any): set this to use the provided array. Used during initialization, and when restoring the updater state (after serialization, for example)

Specified by:

setStateViewArray in interface GradientUpdater

Parameters:

viewArray - Array (that is a view of a larger array) to use for the state.

initialize - If true: the updater must initialize the view array. If false: no change to view array contents

update

public void update(Object... args)

Description copied from interface: GradientUpdater

update(learningRate,momentum)

Specified by:

update in interface GradientUpdater

getGradient

public INDArray getGradient(INDArray gradient,
                            int iteration)

Gets feature specific learning rates Adagrad keeps a history of gradients being passed in. Note that each gradient passed in becomes adapted over time, hence the name adagrad

Specified by:

getGradient in interface GradientUpdater

Parameters:

gradient - the gradient to get learning rates for

iteration -

Returns:

the feature specific learning rates

getGradient

public double getGradient(double gradient,
                          int column,
                          int[] shape)

getGradient

public INDArray getGradient(INDArray gradient,
                            int slice,
                            int[] shape)

createSubset

public AdaGrad createSubset(int index)

getAggregator

public GradientUpdaterAggregator getAggregator(boolean addThis)

Description copied from interface: GradientUpdater

Get a GradientUpdaterAggregator. The aggregator is used (typically in distributed learning scenarios) to combine separate GradientUpdater instances for different networks (usually by averaging).

Specified by:

getAggregator in interface GradientUpdater

Parameters:

addThis - If true: return a GradientUpdaterAggregator with the GradientUpdater already added. If false: return an empty (uninitialized) GradientUpdaterAggregator