ai.djl.training.loss

Class Loss

java.lang.Object

ai.djl.training.evaluator.Evaluator

ai.djl.training.loss.Loss

Direct Known Subclasses:

AbstractCompositeLoss, BertMaskedLanguageModelLoss, BertNextSentenceLoss, ElasticNetWeightDecay, HingeLoss, L1Loss, L1WeightDecay, L2Loss, L2WeightDecay, MaskedSoftmaxCrossEntropyLoss, SigmoidBinaryCrossEntropyLoss, SoftmaxCrossEntropyLoss

public abstract class Loss
extends Evaluator

Loss functions (or Cost functions) are used to evaluate the model predictions against true labels for optimization.

Although all evaluators can be used to measure the performance of a model, not all of them are suited to being used by an optimizer. Loss functions are usually non-negative where a larger loss represents worse performance. They are also real-valued to accurately compare models.

When creating a loss function, you should avoid having the loss depend on the batch size. For example, if you have a loss per item in a batch and sum those losses, your loss would be numItemsInBatch*avgLoss. Instead, you should take the mean of those losses to reduce out the batchSize factor. Otherwise, it can make it difficult to tune the learning rate since any change in the batch size would throw it off. If you have a variable batch size, it would be even more difficult.

For more details about the class internals, see Evaluator.

Field Summary

Fields inherited from class ai.djl.training.evaluator.Evaluator

totalInstances

Constructor Summary

Constructors

Constructor and Description
Loss(java.lang.String name) Base class for metric with abstract update methods.

Method Summary

Modifier and Type	Method and Description
void	addAccumulator(java.lang.String key) Adds an accumulator for the results of the evaluation with the given key.
static ElasticNetWeightDecay	elasticNetWeightedDecay(NDList parameters) Returns a new instance of ElasticNetWeightDecay with default weight and name.
static ElasticNetWeightDecay	elasticNetWeightedDecay(java.lang.String name, float weight1, float weight2, NDList parameters) Returns a new instance of ElasticNetWeightDecay.
static ElasticNetWeightDecay	elasticNetWeightedDecay(java.lang.String name, float weight, NDList parameters) Returns a new instance of ElasticNetWeightDecay.
static ElasticNetWeightDecay	elasticNetWeightedDecay(java.lang.String name, NDList parameters) Returns a new instance of ElasticNetWeightDecay with default weight.
float	getAccumulator(java.lang.String key) Returns the accumulated evaluator value.
static HingeLoss	hingeLoss() Returns a new instance of HingeLoss with default arguments.
static HingeLoss	hingeLoss(java.lang.String name) Returns a new instance of HingeLoss with default arguments.
static HingeLoss	hingeLoss(java.lang.String name, int margin, float weight) Returns a new instance of HingeLoss with the given arguments.
static L1Loss	l1Loss() Returns a new instance of L1Loss with default weight and batch axis.
static L1Loss	l1Loss(java.lang.String name) Returns a new instance of L1Loss with default weight and batch axis.
static L1Loss	l1Loss(java.lang.String name, float weight) Returns a new instance of L1Loss with given weight and batch axis.
static L1WeightDecay	l1WeightedDecay(NDList parameters) Returns a new instance of L1WeightDecay with default weight and name.
static L1WeightDecay	l1WeightedDecay(java.lang.String name, float weight, NDList parameters) Returns a new instance of L1WeightDecay.
static L1WeightDecay	l1WeightedDecay(java.lang.String name, NDList parameters) Returns a new instance of L1WeightDecay with default weight.
static L2Loss	l2Loss() Returns a new instance of L2Loss with default weight and batch axis.
static L2Loss	l2Loss(java.lang.String name) Returns a new instance of L2Loss with default weight and batch axis.
static L2Loss	l2Loss(java.lang.String name, float weight) Returns a new instance of L2Loss with given weight and batch axis.
static L2WeightDecay	l2WeightedDecay(NDList parameters) Returns a new instance of L2WeightDecay with default weight and name.
static L2WeightDecay	l2WeightedDecay(java.lang.String name, float weight, NDList parameters) Returns a new instance of L2WeightDecay.
static L2WeightDecay	l2WeightedDecay(java.lang.String name, NDList parameters) Returns a new instance of L2WeightDecay with default weight.
static MaskedSoftmaxCrossEntropyLoss	maskedSoftmaxCrossEntropyLoss() Returns a new instance of MaskedSoftmaxCrossEntropyLoss with default arguments.
static MaskedSoftmaxCrossEntropyLoss	maskedSoftmaxCrossEntropyLoss(java.lang.String name) Returns a new instance of MaskedSoftmaxCrossEntropyLoss with default arguments.
static MaskedSoftmaxCrossEntropyLoss	maskedSoftmaxCrossEntropyLoss(java.lang.String name, float weight, int classAxis, boolean sparseLabel, boolean fromLogit) Returns a new instance of MaskedSoftmaxCrossEntropyLoss with the given arguments.
void	resetAccumulator(java.lang.String key) Resets the evaluator value with the given key.
static SigmoidBinaryCrossEntropyLoss	sigmoidBinaryCrossEntropyLoss() Returns a new instance of SigmoidBinaryCrossEntropyLoss with default arguments.
static SigmoidBinaryCrossEntropyLoss	sigmoidBinaryCrossEntropyLoss(java.lang.String name) Returns a new instance of SigmoidBinaryCrossEntropyLoss with default arguments.
static SigmoidBinaryCrossEntropyLoss	sigmoidBinaryCrossEntropyLoss(java.lang.String name, float weight, boolean fromSigmoid) Returns a new instance of SigmoidBinaryCrossEntropyLoss with the given arguments.
static SoftmaxCrossEntropyLoss	softmaxCrossEntropyLoss() Returns a new instance of SoftmaxCrossEntropyLoss with default arguments.
static SoftmaxCrossEntropyLoss	softmaxCrossEntropyLoss(java.lang.String name) Returns a new instance of SoftmaxCrossEntropyLoss with default arguments.
static SoftmaxCrossEntropyLoss	softmaxCrossEntropyLoss(java.lang.String name, float weight, int classAxis, boolean sparseLabel, boolean fromLogit) Returns a new instance of SoftmaxCrossEntropyLoss with the given arguments.
void	updateAccumulator(java.lang.String key, NDList labels, NDList predictions) Updates the evaluator with the given key based on a NDList of labels and predictions.

Methods inherited from class ai.djl.training.evaluator.Evaluator

checkLabelShapes, checkLabelShapes, evaluate, getName

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Loss

public Loss(java.lang.String name)

Base class for metric with abstract update methods.

Parameters:

name - The display name of the Loss

Method Detail

l1Loss

public static L1Loss l1Loss()

Returns a new instance of L1Loss with default weight and batch axis.

Returns:

a new instance of L1Loss

l1Loss

public static L1Loss l1Loss(java.lang.String name)

Returns a new instance of L1Loss with default weight and batch axis.

Parameters:

name - the name of the loss

Returns:

a new instance of L1Loss

l1Loss

public static L1Loss l1Loss(java.lang.String name,
                            float weight)

Returns a new instance of L1Loss with given weight and batch axis.

Parameters:

name - the name of the loss

weight - the weight to apply on loss value, default 1

Returns:

a new instance of L1Loss

l2Loss

public static L2Loss l2Loss()

Returns a new instance of L2Loss with default weight and batch axis.

Returns:

a new instance of L2Loss

l2Loss

public static L2Loss l2Loss(java.lang.String name)

Returns a new instance of L2Loss with default weight and batch axis.

Parameters:

name - the name of the loss

Returns:

a new instance of L2Loss

l2Loss

public static L2Loss l2Loss(java.lang.String name,
                            float weight)

Returns a new instance of L2Loss with given weight and batch axis.

Parameters:

name - the name of the loss

weight - the weight to apply on loss value, default 1

Returns:

a new instance of L2Loss

sigmoidBinaryCrossEntropyLoss

public static SigmoidBinaryCrossEntropyLoss sigmoidBinaryCrossEntropyLoss()

Returns a new instance of SigmoidBinaryCrossEntropyLoss with default arguments.

Returns:

a new instance of SigmoidBinaryCrossEntropyLoss

sigmoidBinaryCrossEntropyLoss

public static SigmoidBinaryCrossEntropyLoss sigmoidBinaryCrossEntropyLoss(java.lang.String name)

Returns a new instance of SigmoidBinaryCrossEntropyLoss with default arguments.

Parameters:

name - the name of the loss

Returns:

a new instance of SigmoidBinaryCrossEntropyLoss

sigmoidBinaryCrossEntropyLoss

public static SigmoidBinaryCrossEntropyLoss sigmoidBinaryCrossEntropyLoss(java.lang.String name,
                                                                          float weight,
                                                                          boolean fromSigmoid)

Returns a new instance of SigmoidBinaryCrossEntropyLoss with the given arguments.

Parameters:

name - the name of the loss

weight - the weight to apply on the loss value, default 1

fromSigmoid - whether the input is from the output of sigmoid, default false

Returns:

a new instance of SigmoidBinaryCrossEntropyLoss

softmaxCrossEntropyLoss

public static SoftmaxCrossEntropyLoss softmaxCrossEntropyLoss()

Returns a new instance of SoftmaxCrossEntropyLoss with default arguments.

Returns:

a new instance of SoftmaxCrossEntropyLoss

softmaxCrossEntropyLoss

public static SoftmaxCrossEntropyLoss softmaxCrossEntropyLoss(java.lang.String name)

Returns a new instance of SoftmaxCrossEntropyLoss with default arguments.

Parameters:

name - the name of the loss

Returns:

a new instance of SoftmaxCrossEntropyLoss

softmaxCrossEntropyLoss

public static SoftmaxCrossEntropyLoss softmaxCrossEntropyLoss(java.lang.String name,
                                                              float weight,
                                                              int classAxis,
                                                              boolean sparseLabel,
                                                              boolean fromLogit)

Returns a new instance of SoftmaxCrossEntropyLoss with the given arguments.

Parameters:

name - the name of the loss

weight - the weight to apply on the loss value, default 1

classAxis - the axis that represents the class probabilities, default -1

sparseLabel - whether labels are integer array or probabilities, default true

fromLogit - whether labels are log probabilities or un-normalized numbers

Returns:

a new instance of SoftmaxCrossEntropyLoss

maskedSoftmaxCrossEntropyLoss

public static MaskedSoftmaxCrossEntropyLoss maskedSoftmaxCrossEntropyLoss()

Returns a new instance of MaskedSoftmaxCrossEntropyLoss with default arguments.

Returns:

a new instance of MaskedSoftmaxCrossEntropyLoss

maskedSoftmaxCrossEntropyLoss

public static MaskedSoftmaxCrossEntropyLoss maskedSoftmaxCrossEntropyLoss(java.lang.String name)

Returns a new instance of MaskedSoftmaxCrossEntropyLoss with default arguments.

Parameters:

name - the name of the loss

Returns:

a new instance of MaskedSoftmaxCrossEntropyLoss

maskedSoftmaxCrossEntropyLoss

public static MaskedSoftmaxCrossEntropyLoss maskedSoftmaxCrossEntropyLoss(java.lang.String name,
                                                                          float weight,
                                                                          int classAxis,
                                                                          boolean sparseLabel,
                                                                          boolean fromLogit)

Returns a new instance of MaskedSoftmaxCrossEntropyLoss with the given arguments.

Parameters:

name - the name of the loss

weight - the weight to apply on the loss value, default 1

classAxis - the axis that represents the class probabilities, default -1

sparseLabel - whether labels are integer array or probabilities, default true

fromLogit - whether labels are log probabilities or un-normalized numbers

Returns:

a new instance of MaskedSoftmaxCrossEntropyLoss

hingeLoss

public static HingeLoss hingeLoss()

Returns a new instance of HingeLoss with default arguments.

Returns:

a new instance of HingeLoss

hingeLoss

public static HingeLoss hingeLoss(java.lang.String name)

Returns a new instance of HingeLoss with default arguments.

Parameters:

name - the name of the loss

Returns:

a new instance of HingeLoss

hingeLoss

public static HingeLoss hingeLoss(java.lang.String name,
                                  int margin,
                                  float weight)

Returns a new instance of HingeLoss with the given arguments.

Parameters:

name - the name of the loss

margin - the margin in hinge loss. Defaults to 1.0

weight - the weight to apply on loss value, default 1

Returns:

a new instance of HingeLoss

l1WeightedDecay

public static L1WeightDecay l1WeightedDecay(NDList parameters)

Returns a new instance of L1WeightDecay with default weight and name.

Parameters:

parameters - holds the model weights that will be penalized

Returns:

a new instance of L1WeightDecay

l1WeightedDecay

public static L1WeightDecay l1WeightedDecay(java.lang.String name,
                                            NDList parameters)

Returns a new instance of L1WeightDecay with default weight.

Parameters:

name - the name of the weight decay

parameters - holds the model weights that will be penalized

Returns:

a new instance of L1WeightDecay

l1WeightedDecay

public static L1WeightDecay l1WeightedDecay(java.lang.String name,
                                            float weight,
                                            NDList parameters)

Returns a new instance of L1WeightDecay.

Parameters:

name - the name of the weight decay

weight - the weight to apply on weight decay value, default 1

parameters - holds the model weights that will be penalized

Returns:

a new instance of L1WeightDecay

l2WeightedDecay

public static L2WeightDecay l2WeightedDecay(NDList parameters)

Returns a new instance of L2WeightDecay with default weight and name.

Parameters:

parameters - holds the model weights that will be penalized

Returns:

a new instance of L2WeightDecay

l2WeightedDecay

public static L2WeightDecay l2WeightedDecay(java.lang.String name,
                                            NDList parameters)

Returns a new instance of L2WeightDecay with default weight.

Parameters:

name - the name of the weight decay

parameters - holds the model weights that will be penalized

Returns:

a new instance of L2WeightDecay

l2WeightedDecay

public static L2WeightDecay l2WeightedDecay(java.lang.String name,
                                            float weight,
                                            NDList parameters)

Returns a new instance of L2WeightDecay.

Parameters:

name - the name of the weight decay

weight - the weight to apply on weight decay value, default 1

parameters - holds the model weights that will be penalized

Returns:

a new instance of L2WeightDecay

elasticNetWeightedDecay

public static ElasticNetWeightDecay elasticNetWeightedDecay(NDList parameters)

Returns a new instance of ElasticNetWeightDecay with default weight and name.

Parameters:

parameters - holds the model weights that will be penalized

Returns:

a new instance of ElasticNetWeightDecay

elasticNetWeightedDecay

public static ElasticNetWeightDecay elasticNetWeightedDecay(java.lang.String name,
                                                            NDList parameters)

Returns a new instance of ElasticNetWeightDecay with default weight.

Parameters:

name - the name of the weight decay

parameters - holds the model weights that will be penalized

Returns:

a new instance of ElasticNetWeightDecay

elasticNetWeightedDecay

public static ElasticNetWeightDecay elasticNetWeightedDecay(java.lang.String name,
                                                            float weight,
                                                            NDList parameters)

Returns a new instance of ElasticNetWeightDecay.

Parameters:

name - the name of the weight decay

weight - the weight to apply on weight decay values, default 1

parameters - holds the model weights that will be penalized

Returns:

a new instance of ElasticNetWeightDecay

elasticNetWeightedDecay

public static ElasticNetWeightDecay elasticNetWeightedDecay(java.lang.String name,
                                                            float weight1,
                                                            float weight2,
                                                            NDList parameters)

Returns a new instance of ElasticNetWeightDecay.

Parameters:

name - the name of the weight decay

weight1 - the weight to apply on weight decay L1 value, default 1

weight2 - the weight to apply on weight decay L2 value, default 1

parameters - holds the model weights that will be penalized

Returns:

a new instance of ElasticNetWeightDecay

addAccumulator

public void addAccumulator(java.lang.String key)

Adds an accumulator for the results of the evaluation with the given key.

Specified by:

addAccumulator in class Evaluator

Parameters:

key - the key for the new accumulator

updateAccumulator

public void updateAccumulator(java.lang.String key,
                              NDList labels,
                              NDList predictions)

Updates the evaluator with the given key based on a NDList of labels and predictions.

This is a synchronized operation. You should only call it at the end of a batch or epoch.

Specified by:

updateAccumulator in class Evaluator

Parameters:

key - the key of the accumulator to update

labels - a NDList of labels

predictions - a NDList of predictions

resetAccumulator

public void resetAccumulator(java.lang.String key)

Resets the evaluator value with the given key.

Specified by:

resetAccumulator in class Evaluator

Parameters:

key - the key of the accumulator to reset

getAccumulator

public float getAccumulator(java.lang.String key)

Returns the accumulated evaluator value.

Specified by:

getAccumulator in class Evaluator

Parameters:

key - the key of the accumulator to get

Returns:

the accumulated value