org.nd4j.linalg.factory.ops

Class NDLoss

java.lang.Object

org.nd4j.linalg.factory.ops.NDLoss

public class NDLoss
extends Object

Constructor Summary

Constructors

Constructor and Description
NDLoss()

Method Summary

Modifier and Type	Method and Description
INDArray	absoluteDifference(INDArray label, INDArray predictions, INDArray weights) Absolute difference loss: sum_i abs( label[i] - predictions[i] )
INDArray	absoluteDifference(INDArray label, INDArray predictions, INDArray weights, LossReduce lossReduce) Absolute difference loss: sum_i abs( label[i] - predictions[i] )
INDArray	cosineDistance(INDArray label, INDArray predictions, INDArray weights, int dimension) Cosine distance loss: 1 - cosineSimilarity(x,y) or 1 - sum_i label[i] * prediction[i], which is equivalent to cosine distance when both the predictions and labels are normalized. Note: This loss function assumes that both the predictions and labels are normalized to have unit l2 norm. If this is not the case, you should normalize them first by dividing by norm2(String, SDVariable, boolean, int...) along the cosine distance dimension (with keepDims=true).
INDArray	cosineDistance(INDArray label, INDArray predictions, INDArray weights, LossReduce lossReduce, int dimension) Cosine distance loss: 1 - cosineSimilarity(x,y) or 1 - sum_i label[i] * prediction[i], which is equivalent to cosine distance when both the predictions and labels are normalized. Note: This loss function assumes that both the predictions and labels are normalized to have unit l2 norm. If this is not the case, you should normalize them first by dividing by norm2(String, SDVariable, boolean, int...) along the cosine distance dimension (with keepDims=true).
INDArray	hingeLoss(INDArray label, INDArray predictions, INDArray weights) Hinge loss: a loss function used for training classifiers. Implements L = max(0, 1 - t * predictions) where t is the label values after internally converting to {-1,1} from the user specified {0,1}.
INDArray	hingeLoss(INDArray label, INDArray predictions, INDArray weights, LossReduce lossReduce) Hinge loss: a loss function used for training classifiers. Implements L = max(0, 1 - t * predictions) where t is the label values after internally converting to {-1,1} from the user specified {0,1}.
INDArray	huberLoss(INDArray label, INDArray predictions, INDArray weights, double delta) Huber loss function, used for robust regression.
INDArray	huberLoss(INDArray label, INDArray predictions, INDArray weights, LossReduce lossReduce, double delta) Huber loss function, used for robust regression.
INDArray	l2Loss(INDArray var) L2 loss: 1/2 * sum(x^2)
INDArray	logLoss(INDArray label, INDArray predictions) Log loss, i.e., binary cross entropy loss, usually used for binary multi-label classification.
INDArray	logLoss(INDArray label, INDArray predictions, INDArray weights, LossReduce lossReduce, double epsilon) Log loss, i.e., binary cross entropy loss, usually used for binary multi-label classification.
INDArray	logPoisson(INDArray label, INDArray predictions, INDArray weights, boolean full) Log poisson loss: a loss function used for training classifiers. Implements L = exp(c) - z * c where c is log(predictions) and z is labels.
INDArray	logPoisson(INDArray label, INDArray predictions, INDArray weights, LossReduce lossReduce, boolean full) Log poisson loss: a loss function used for training classifiers. Implements L = exp(c) - z * c where c is log(predictions) and z is labels.
INDArray	meanPairwiseSquaredError(INDArray label, INDArray predictions, INDArray weights) Mean pairwise squared error. MPWSE loss calculates the difference between pairs of consecutive elements in the predictions and labels arrays. For example, if predictions = [p0, p1, p2] and labels are [l0, l1, l2] then MPWSE is: [((p0-p1) - (l0-l1))^2 + ((p0-p2) - (l0-l2))^2 + ((p1-p2) - (l1-l2))^2] / 3
INDArray	meanPairwiseSquaredError(INDArray label, INDArray predictions, INDArray weights, LossReduce lossReduce) Mean pairwise squared error. MPWSE loss calculates the difference between pairs of consecutive elements in the predictions and labels arrays. For example, if predictions = [p0, p1, p2] and labels are [l0, l1, l2] then MPWSE is: [((p0-p1) - (l0-l1))^2 + ((p0-p2) - (l0-l2))^2 + ((p1-p2) - (l1-l2))^2] / 3
INDArray	meanSquaredError(INDArray label, INDArray predictions, INDArray weights) Mean squared error loss function.
INDArray	meanSquaredError(INDArray label, INDArray predictions, INDArray weights, LossReduce lossReduce) Mean squared error loss function.
INDArray	sigmoidCrossEntropy(INDArray label, INDArray predictionLogits, INDArray weights) Sigmoid cross entropy: applies the sigmoid activation function on the input logits (input "pre-sigmoid preductions") and implements the binary cross entropy loss function.
INDArray	sigmoidCrossEntropy(INDArray label, INDArray predictionLogits, INDArray weights, LossReduce lossReduce, double labelSmoothing) Sigmoid cross entropy: applies the sigmoid activation function on the input logits (input "pre-sigmoid preductions") and implements the binary cross entropy loss function.
INDArray	softmaxCrossEntropy(INDArray oneHotLabels, INDArray logitPredictions, INDArray weights) Applies the softmax activation function to the input, then implement multi-class cross entropy: -sum_classes label[i] * log(p[c]) where p = softmax(logits) If LossReduce.NONE is used, returned shape is [numExamples] out for [numExamples, numClasses] predicitons/labels; otherwise, the output is a scalar.
INDArray	softmaxCrossEntropy(INDArray oneHotLabels, INDArray logitPredictions, INDArray weights, LossReduce lossReduce, double labelSmoothing) Applies the softmax activation function to the input, then implement multi-class cross entropy: -sum_classes label[i] * log(p[c]) where p = softmax(logits) If LossReduce.NONE is used, returned shape is [numExamples] out for [numExamples, numClasses] predicitons/labels; otherwise, the output is a scalar.
INDArray	sparseSoftmaxCrossEntropy(INDArray logits, INDArray labels) As per softmaxCrossEntropy(String, SDVariable, SDVariable, LossReduce) but the labels variable is represented as an integer array instead of the equivalent one-hot array. i.e., if logits are rank N, then labels have rank N-1
INDArray	weightedCrossEntropyWithLogits(INDArray targets, INDArray inputs, INDArray weights) Weighted cross entropy loss with logits

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

NDLoss

public NDLoss()

Method Detail

absoluteDifference

public INDArray absoluteDifference(INDArray label,
                                   INDArray predictions,
                                   INDArray weights,
                                   LossReduce lossReduce)

Absolute difference loss: sum_i abs( label[i] - predictions[i] )

Parameters:

label - Label array (NUMERIC type)

predictions - Predictions array (NUMERIC type)

weights - Weights array. May be null. If null, a weight of 1.0 is used (NUMERIC type)

lossReduce - Reduction type for the loss. See LossReduce for more details. Default: LossReduce.MEAN_BY_NONZERO_WEIGHT_COUNT

Returns:

output loss variable (NUMERIC type)

absoluteDifference

public INDArray absoluteDifference(INDArray label,
                                   INDArray predictions,
                                   INDArray weights)

Absolute difference loss: sum_i abs( label[i] - predictions[i] )

Parameters:

label - Label array (NUMERIC type)

predictions - Predictions array (NUMERIC type)

weights - Weights array. May be null. If null, a weight of 1.0 is used (NUMERIC type)

Returns:

output loss variable (NUMERIC type)

cosineDistance

public INDArray cosineDistance(INDArray label,
                               INDArray predictions,
                               INDArray weights,
                               LossReduce lossReduce,
                               int dimension)

Cosine distance loss: 1 - cosineSimilarity(x,y) or 1 - sum_i label[i] * prediction[i], which is
equivalent to cosine distance when both the predictions and labels are normalized.
Note: This loss function assumes that both the predictions and labels are normalized to have unit l2 norm.
If this is not the case, you should normalize them first by dividing by norm2(String, SDVariable, boolean, int...)
along the cosine distance dimension (with keepDims=true).

Parameters:

label - Label array (NUMERIC type)

predictions - Predictions array (NUMERIC type)

weights - Weights array. May be null. If null, a weight of 1.0 is use (NUMERIC type)

lossReduce - Reduction type for the loss. See LossReduce for more details. Default: LossReduce.MEAN_BY_NONZERO_WEIGHT_COUNT

dimension - Dimension to perform the cosine distance over

Returns:

output Cosine distance loss (NUMERIC type)

cosineDistance

public INDArray cosineDistance(INDArray label,
                               INDArray predictions,
                               INDArray weights,
                               int dimension)

Cosine distance loss: 1 - cosineSimilarity(x,y) or 1 - sum_i label[i] * prediction[i], which is
equivalent to cosine distance when both the predictions and labels are normalized.
Note: This loss function assumes that both the predictions and labels are normalized to have unit l2 norm.
If this is not the case, you should normalize them first by dividing by norm2(String, SDVariable, boolean, int...)
along the cosine distance dimension (with keepDims=true).

Parameters:

label - Label array (NUMERIC type)

predictions - Predictions array (NUMERIC type)

weights - Weights array. May be null. If null, a weight of 1.0 is use (NUMERIC type)

dimension - Dimension to perform the cosine distance over

Returns:

output Cosine distance loss (NUMERIC type)

hingeLoss

public INDArray hingeLoss(INDArray label,
                          INDArray predictions,
                          INDArray weights,
                          LossReduce lossReduce)

Hinge loss: a loss function used for training classifiers.
Implements L = max(0, 1 - t * predictions) where t is the label values after internally converting to {-1,1}
from the user specified {0,1}. Note that Labels should be provided with values {0,1}.

Parameters:

label - Label array. Each value should be 0.0 or 1.0 (internally -1 to 1 is used) (NUMERIC type)

predictions - Predictions array (NUMERIC type)

weights - Weights array. May be null. If null, a weight of 1.0 is used (NUMERIC type)

lossReduce - Reduction type for the loss. See LossReduce for more details. Default: LossReduce.MEAN_BY_NONZERO_WEIGHT_COUNT

Returns:

output Loss variable (NUMERIC type)

hingeLoss

public INDArray hingeLoss(INDArray label,
                          INDArray predictions,
                          INDArray weights)

Hinge loss: a loss function used for training classifiers.
Implements L = max(0, 1 - t * predictions) where t is the label values after internally converting to {-1,1}
from the user specified {0,1}. Note that Labels should be provided with values {0,1}.

Parameters:

label - Label array. Each value should be 0.0 or 1.0 (internally -1 to 1 is used) (NUMERIC type)

predictions - Predictions array (NUMERIC type)

weights - Weights array. May be null. If null, a weight of 1.0 is used (NUMERIC type)

Returns:

output Loss variable (NUMERIC type)

huberLoss

public INDArray huberLoss(INDArray label,
                          INDArray predictions,
                          INDArray weights,
                          LossReduce lossReduce,
                          double delta)

Huber loss function, used for robust regression. It is similar both squared error loss and absolute difference loss,
though is less sensitive to outliers than squared error.
Huber loss implements:

  L = 0.5 * (label[i] - predictions[i])^2 if abs(label[i] - predictions[i]) < delta

  L = delta * abs(label[i] - predictions[i]) - 0.5 * delta^2 otherwise

 

Parameters:

label - Label array (NUMERIC type)

predictions - Predictions array (NUMERIC type)

weights - Weights array. May be null. If null, a weight of 1.0 is used (NUMERIC type)

lossReduce - Reduction type for the loss. See LossReduce for more details. Default: LossReduce.MEAN_BY_NONZERO_WEIGHT_COUNT

delta - Loss function delta value

Returns:

output Huber loss (NUMERIC type)

huberLoss

public INDArray huberLoss(INDArray label,
                          INDArray predictions,
                          INDArray weights,
                          double delta)

Huber loss function, used for robust regression. It is similar both squared error loss and absolute difference loss,
though is less sensitive to outliers than squared error.
Huber loss implements:

  L = 0.5 * (label[i] - predictions[i])^2 if abs(label[i] - predictions[i]) < delta

  L = delta * abs(label[i] - predictions[i]) - 0.5 * delta^2 otherwise

 

Parameters:

label - Label array (NUMERIC type)

predictions - Predictions array (NUMERIC type)

weights - Weights array. May be null. If null, a weight of 1.0 is used (NUMERIC type)

delta - Loss function delta value

Returns:

output Huber loss (NUMERIC type)

l2Loss

public INDArray l2Loss(INDArray var)

L2 loss: 1/2 * sum(x^2)

Parameters:

var - Variable to calculate L2 loss of (NUMERIC type)

Returns:

output L2 loss (NUMERIC type)

logLoss

public INDArray logLoss(INDArray label,
                        INDArray predictions,
                        INDArray weights,
                        LossReduce lossReduce,
                        double epsilon)

Log loss, i.e., binary cross entropy loss, usually used for binary multi-label classification. Implements:
-1/numExamples * sum_i (labels[i] * log(predictions[i] + epsilon) + (1-labels[i]) * log(1-predictions[i] + epsilon))

Parameters:

label - Label array (NUMERIC type)

predictions - Predictions array (NUMERIC type)

weights - Weights array. May be null. If null, a weight of 1.0 is used (NUMERIC type)

lossReduce - Reduction type for the loss. See LossReduce for more details. Default: LossReduce.MEAN_BY_NONZERO_WEIGHT_COUNT

epsilon - epsilon

Returns:

output Log loss (NUMERIC type)

logLoss

public INDArray logLoss(INDArray label,
                        INDArray predictions)

Log loss, i.e., binary cross entropy loss, usually used for binary multi-label classification. Implements:
-1/numExamples * sum_i (labels[i] * log(predictions[i] + epsilon) + (1-labels[i]) * log(1-predictions[i] + epsilon))

Parameters:

label - Label array (NUMERIC type)

predictions - Predictions array (NUMERIC type)

Returns:

output Log loss (NUMERIC type)

logPoisson

public INDArray logPoisson(INDArray label,
                           INDArray predictions,
                           INDArray weights,
                           LossReduce lossReduce,
                           boolean full)

Log poisson loss: a loss function used for training classifiers.
Implements L = exp(c) - z * c where c is log(predictions) and z is labels.

Parameters:

label - Label array. Each value should be 0.0 or 1.0 (NUMERIC type)

predictions - Predictions array (has to be log(x) of actual predictions) (NUMERIC type)

weights - Weights array. May be null. If null, a weight of 1.0 is used (NUMERIC type)

lossReduce - Reduction type for the loss. See LossReduce for more details. Default: LossReduce.MEAN_BY_NONZERO_WEIGHT_COUNT

full - Boolean flag. true for logPoissonFull, false for logPoisson

Returns:

output Loss variable (NUMERIC type)

logPoisson

public INDArray logPoisson(INDArray label,
                           INDArray predictions,
                           INDArray weights,
                           boolean full)

Log poisson loss: a loss function used for training classifiers.
Implements L = exp(c) - z * c where c is log(predictions) and z is labels.

Parameters:

label - Label array. Each value should be 0.0 or 1.0 (NUMERIC type)

predictions - Predictions array (has to be log(x) of actual predictions) (NUMERIC type)

weights - Weights array. May be null. If null, a weight of 1.0 is used (NUMERIC type)

full - Boolean flag. true for logPoissonFull, false for logPoisson

Returns:

output Loss variable (NUMERIC type)

meanPairwiseSquaredError

public INDArray meanPairwiseSquaredError(INDArray label,
                                         INDArray predictions,
                                         INDArray weights,
                                         LossReduce lossReduce)

Mean pairwise squared error.
MPWSE loss calculates the difference between pairs of consecutive elements in the predictions and labels arrays.
For example, if predictions = [p0, p1, p2] and labels are [l0, l1, l2] then MPWSE is:
[((p0-p1) - (l0-l1))^2 + ((p0-p2) - (l0-l2))^2 + ((p1-p2) - (l1-l2))^2] / 3

Parameters:

label - Label array (NUMERIC type)

predictions - Predictions array (NUMERIC type)

weights - Weights array. May be null. If null, a weight of 1.0 is used. Must be either null, scalar, or have shape [batchSize] (NUMERIC type)

lossReduce - Reduction type for the loss. See LossReduce for more details. Default: LossReduce.MEAN_BY_NONZERO_WEIGHT_COUNT

Returns:

output Loss variable, scalar output (NUMERIC type)

meanPairwiseSquaredError

public INDArray meanPairwiseSquaredError(INDArray label,
                                         INDArray predictions,
                                         INDArray weights)

Mean pairwise squared error.
MPWSE loss calculates the difference between pairs of consecutive elements in the predictions and labels arrays.
For example, if predictions = [p0, p1, p2] and labels are [l0, l1, l2] then MPWSE is:
[((p0-p1) - (l0-l1))^2 + ((p0-p2) - (l0-l2))^2 + ((p1-p2) - (l1-l2))^2] / 3

Parameters:

label - Label array (NUMERIC type)

predictions - Predictions array (NUMERIC type)

weights - Weights array. May be null. If null, a weight of 1.0 is used. Must be either null, scalar, or have shape [batchSize] (NUMERIC type)

Returns:

output Loss variable, scalar output (NUMERIC type)

meanSquaredError

public INDArray meanSquaredError(INDArray label,
                                 INDArray predictions,
                                 INDArray weights,
                                 LossReduce lossReduce)

Mean squared error loss function. Implements (label[i] - prediction[i])^2 - i.e., squared error on a per-element basis.
When averaged (using LossReduce.MEAN_BY_WEIGHT or LossReduce.MEAN_BY_NONZERO_WEIGHT_COUNT (the default))
this is the mean squared error loss function.

Parameters:

label - Label array (NUMERIC type)

predictions - Predictions array (NUMERIC type)

weights - Weights array. May be null. If null, a weight of 1.0 is used (NUMERIC type)

lossReduce - Reduction type for the loss. See LossReduce for more details. Default: LossReduce.MEAN_BY_NONZERO_WEIGHT_COUNT

Returns:

output Loss variable (NUMERIC type)

meanSquaredError

public INDArray meanSquaredError(INDArray label,
                                 INDArray predictions,
                                 INDArray weights)

Mean squared error loss function. Implements (label[i] - prediction[i])^2 - i.e., squared error on a per-element basis.
When averaged (using LossReduce.MEAN_BY_WEIGHT or LossReduce.MEAN_BY_NONZERO_WEIGHT_COUNT (the default))
this is the mean squared error loss function.

Parameters:

label - Label array (NUMERIC type)

predictions - Predictions array (NUMERIC type)

weights - Weights array. May be null. If null, a weight of 1.0 is used (NUMERIC type)

Returns:

output Loss variable (NUMERIC type)

sigmoidCrossEntropy

public INDArray sigmoidCrossEntropy(INDArray label,
                                    INDArray predictionLogits,
                                    INDArray weights,
                                    LossReduce lossReduce,
                                    double labelSmoothing)

Sigmoid cross entropy: applies the sigmoid activation function on the input logits (input "pre-sigmoid preductions")
and implements the binary cross entropy loss function. This implementation is numerically more stable than using
standard (but separate) sigmoid activation function and log loss (binary cross entropy) loss function.
Implements:
-1/numExamples * sum_i (labels[i] * log(sigmoid(logits[i])) + (1-labels[i]) * log(1-sigmoid(logits[i])))
though this is done in a mathematically equivalent but more numerical stable form.

When label smoothing is > 0, the following label smoothing is used:

  numClasses = labels.size(1);<br>
 label = (1.0 - labelSmoothing) * label + 0.5 * labelSmoothing

 

Parameters:

label - Label array (NUMERIC type)

predictionLogits - Predictions array (NUMERIC type)

weights - Weights array. May be null. If null, a weight of 1.0 is used (NUMERIC type)

lossReduce - Reduction type for the loss. See LossReduce for more details. Default: LossReduce.MEAN_BY_NONZERO_WEIGHT_COUNT

labelSmoothing - Label smoothing value. Default value: 0

Returns:

output Loss variable (NUMERIC type)

sigmoidCrossEntropy

public INDArray sigmoidCrossEntropy(INDArray label,
                                    INDArray predictionLogits,
                                    INDArray weights)

Sigmoid cross entropy: applies the sigmoid activation function on the input logits (input "pre-sigmoid preductions")
and implements the binary cross entropy loss function. This implementation is numerically more stable than using
standard (but separate) sigmoid activation function and log loss (binary cross entropy) loss function.
Implements:
-1/numExamples * sum_i (labels[i] * log(sigmoid(logits[i])) + (1-labels[i]) * log(1-sigmoid(logits[i])))
though this is done in a mathematically equivalent but more numerical stable form.

When label smoothing is > 0, the following label smoothing is used:

  numClasses = labels.size(1);<br>
 label = (1.0 - labelSmoothing) * label + 0.5 * labelSmoothing

 

Parameters:

label - Label array (NUMERIC type)

predictionLogits - Predictions array (NUMERIC type)

weights - Weights array. May be null. If null, a weight of 1.0 is used (NUMERIC type)

Returns:

output Loss variable (NUMERIC type)

softmaxCrossEntropy

public INDArray softmaxCrossEntropy(INDArray oneHotLabels,
                                    INDArray logitPredictions,
                                    INDArray weights,
                                    LossReduce lossReduce,
                                    double labelSmoothing)

Applies the softmax activation function to the input, then implement multi-class cross entropy:
-sum_classes label[i] * log(p[c]) where p = softmax(logits)
If LossReduce.NONE is used, returned shape is [numExamples] out for [numExamples, numClasses] predicitons/labels;
otherwise, the output is a scalar.

When label smoothing is > 0, the following label smoothing is used:

  numClasses = labels.size(1);<br>
 oneHotLabel = (1.0 - labelSmoothing) * oneHotLabels + labelSmoothing/numClasses

 

Parameters:

oneHotLabels - Label array. Should be one-hot per example and same shape as predictions (for example, [mb, nOut]) (NUMERIC type)

logitPredictions - Predictions array (pre-softmax) (NUMERIC type)

weights - Weights array. May be null. If null, a weight of 1.0 is used (NUMERIC type)

lossReduce - Reduction type for the loss. See LossReduce for more details. Default: LossReduce.MEAN_BY_NONZERO_WEIGHT_COUNT

labelSmoothing - Label smoothing value. Default value: 0

Returns:

output Loss variable (NUMERIC type)

softmaxCrossEntropy

public INDArray softmaxCrossEntropy(INDArray oneHotLabels,
                                    INDArray logitPredictions,
                                    INDArray weights)

Applies the softmax activation function to the input, then implement multi-class cross entropy:
-sum_classes label[i] * log(p[c]) where p = softmax(logits)
If LossReduce.NONE is used, returned shape is [numExamples] out for [numExamples, numClasses] predicitons/labels;
otherwise, the output is a scalar.

When label smoothing is > 0, the following label smoothing is used:

  numClasses = labels.size(1);<br>
 oneHotLabel = (1.0 - labelSmoothing) * oneHotLabels + labelSmoothing/numClasses

 

Parameters:

oneHotLabels - Label array. Should be one-hot per example and same shape as predictions (for example, [mb, nOut]) (NUMERIC type)

logitPredictions - Predictions array (pre-softmax) (NUMERIC type)

weights - Weights array. May be null. If null, a weight of 1.0 is used (NUMERIC type)

Returns:

output Loss variable (NUMERIC type)

sparseSoftmaxCrossEntropy

public INDArray sparseSoftmaxCrossEntropy(INDArray logits,
                                          INDArray labels)

As per softmaxCrossEntropy(String, SDVariable, SDVariable, LossReduce) but the labels variable
is represented as an integer array instead of the equivalent one-hot array.
i.e., if logits are rank N, then labels have rank N-1

Parameters:

logits - Logits array ("pre-softmax activations") (NUMERIC type)

labels - Labels array. Must be an integer type. (INT type)

Returns:

output Softmax cross entropy (NUMERIC type)

weightedCrossEntropyWithLogits

public INDArray weightedCrossEntropyWithLogits(INDArray targets,
                                               INDArray inputs,
                                               INDArray weights)

Weighted cross entropy loss with logits

Parameters:

targets - targets array (NUMERIC type)

inputs - input array (NUMERIC type)

weights - eights array. May be null. If null, a weight of 1.0 is used (NUMERIC type)

Returns:

output Loss variable (NUMERIC type)