org.nd4j.linalg.learning.regularization

Class L1Regularization

java.lang.Object

org.nd4j.linalg.learning.regularization.L1Regularization

All Implemented Interfaces:

Serializable, Regularization

public class L1Regularization
extends Object
implements Regularization

L1 regularization: Implements updating as follows:
L = loss + l1 * sum_i abs(w[i])
{@code w[i] -= updater(gradient[i] + l1 * sign(w[i])) - where sign(w[i]) is +/- 1
Note that L1 regularization is applied before the updater (Adam/Nesterov/etc) is applied.

Author:

Alex Black

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from interface org.nd4j.linalg.learning.regularization.Regularization

Regularization.ApplyStep

Field Summary

Fields

Modifier and Type	Field and Description
protected ISchedule	l1

Constructor Summary

Constructors

Constructor and Description
L1Regularization(double l1)
L1Regularization(ISchedule l1)

Method Summary

Modifier and Type	Method and Description
void	apply(INDArray param, INDArray gradView, double lr, int iteration, int epoch) Apply the regularization by modifying the gradient array in-place
Regularization.ApplyStep	applyStep()
Regularization	clone()
double	score(INDArray param, int iteration, int epoch) Calculate the loss function score component for the regularization. For example, in L2 regularization, this would return L = 0.5 * sum_i param[i]^2 For regularization types that don't have a score component, this method can return 0.

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

l1

protected final ISchedule l1

Constructor Detail

L1Regularization

public L1Regularization(double l1)

Parameters:

l1 - l1 regularization coefficient

L1Regularization

public L1Regularization(@NonNull
                        ISchedule l1)

Parameters:

l1 - L1 regularization coefficient (schedule)

Method Detail

applyStep

public Regularization.ApplyStep applyStep()

Specified by:

applyStep in interface Regularization

Returns:

The step that the regularization should be applied, as defined by Regularization.ApplyStep

apply

public void apply(INDArray param,
                  INDArray gradView,
                  double lr,
                  int iteration,
                  int epoch)

Description copied from interface: Regularization

Apply the regularization by modifying the gradient array in-place

Specified by:

apply in interface Regularization

Parameters:

param - Input array (usually parameters)

gradView - Gradient view array (should be modified/updated). Same shape and type as the input array.

lr - Current learning rate

iteration - Current network training iteration

epoch - Current network training epoch

score

public double score(INDArray param,
                    int iteration,
                    int epoch)

Description copied from interface: Regularization

Calculate the loss function score component for the regularization.
For example, in L2 regularization, this would return L = 0.5 * sum_i param[i]^2
For regularization types that don't have a score component, this method can return 0. However, note that this may make the regularization type not gradient checkable.

Specified by:

score in interface Regularization

Parameters:

param - Input array (usually parameters)

iteration - Current network training iteration

epoch - Current network training epoch

Returns:

Loss function score component based on the input/parameters array

clone

public Regularization clone()

Specified by:

clone in interface Regularization

Overrides:

clone in class Object

Returns:

An independent copy of the regularization instance