org.deeplearning4j.gradientcheck

Class GradientCheckUtil

java.lang.Object

org.deeplearning4j.gradientcheck.GradientCheckUtil

public class GradientCheckUtil
extends Object

A utility for numerically checking gradients.
Basic idea: compare calculated gradients with those calculated numerically, to check implementation of backpropagation gradient calculation.
See:
- http://cs231n.github.io/neural-networks-3/#gradcheck
- http://ufldl.stanford.edu/wiki/index.php/Gradient_checking_and_advanced_optimization
- https://code.google.com/p/cuda-convnet/wiki/CheckingGradients
Is C is cost function, then dC/dw ~= (C(w+epsilon)-C(w-epsilon)) / (2*epsilon).
Method checks gradient calculation for every parameter separately by doing 2 forward pass calculations for each parameter, so can be very time consuming for large networks.

Author:

Alex Black

Method Summary

Modifier and Type	Method and Description
static boolean	checkGradients(ComputationGraph graph, double epsilon, double maxRelError, boolean print, boolean exitOnFirstError, org.nd4j.linalg.api.ndarray.INDArray[] inputs, org.nd4j.linalg.api.ndarray.INDArray[] labels) Check backprop gradients for a ComputationGraph
static boolean	checkGradients(MultiLayerNetwork mln, double epsilon, double maxRelError, boolean print, boolean exitOnFirstError, org.nd4j.linalg.api.ndarray.INDArray input, org.nd4j.linalg.api.ndarray.INDArray labels, boolean useUpdater) Check backprop gradients for a MultiLayerNetwork.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

checkGradients

public static boolean checkGradients(MultiLayerNetwork mln,
                                     double epsilon,
                                     double maxRelError,
                                     boolean print,
                                     boolean exitOnFirstError,
                                     org.nd4j.linalg.api.ndarray.INDArray input,
                                     org.nd4j.linalg.api.ndarray.INDArray labels,
                                     boolean useUpdater)

Check backprop gradients for a MultiLayerNetwork.

Parameters:

mln - MultiLayerNetwork to test. This must be initialized.

epsilon - Usually on the order/ of 1e-4 or so.

maxRelError - Maximum relative error. Usually < 0.01, though maybe more for deep networks

print - Whether to print full pass/failure details for each parameter gradient

exitOnFirstError - If true: return upon first failure. If false: continue checking even if one parameter gradient has failed. Typically use false for debugging, true for unit tests.

input - Input array to use for forward pass. May be mini-batch data.

labels - Labels/targets to use to calculate backprop gradient. May be mini-batch data.

useUpdater - Whether to put the gradient through Updater.update(...). Necessary for testing things like l1 and l2 regularization.

Returns:

true if gradients are passed, false otherwise.

checkGradients

public static boolean checkGradients(ComputationGraph graph,
                                     double epsilon,
                                     double maxRelError,
                                     boolean print,
                                     boolean exitOnFirstError,
                                     org.nd4j.linalg.api.ndarray.INDArray[] inputs,
                                     org.nd4j.linalg.api.ndarray.INDArray[] labels)

Check backprop gradients for a ComputationGraph

Parameters:

graph - ComputationGraph to test. This must be initialized.

epsilon - Usually on the order of 1e-4 or so.

maxRelError - Maximum relative error. Usually < 0.01, though maybe more for deep networks

print - Whether to print full pass/failure details for each parameter gradient

exitOnFirstError - If true: return upon first failure. If false: continue checking even if one parameter gradient has failed. Typically use false for debugging, true for unit tests.

inputs - Input arrays to use for forward pass. May be mini-batch data.

labels - Labels/targets (output) arrays to use to calculate backprop gradient. May be mini-batch data.

Returns:

true if gradients are passed, false otherwise.