Interface | Description |
---|---|
Regularization |
Regularization API for magnitude-based regularization techniques such as:
L1Regularization L2Regularization WeightDecay Implementations should have the following features: 1. |
Class | Description |
---|---|
L1Regularization |
L1 regularization: Implements updating as follows:
L = loss + l1 * sum_i abs(w[i]) {@code w[i] -= updater(gradient[i] + l1 * sign(w[i])) - where sign(w[i]) is +/- 1 Note that L1 regularization is applied before the updater (Adam/Nesterov/etc) is applied. |
L2Regularization |
L2 regularization: very similar to
WeightDecay , but is applied before the updater is applied, not after. |
WeightDecay |
WeightDecay regularization: Updater is not applied to the regularization term gradients, and (optionally) applies the learning rate.
|
Enum | Description |
---|---|
Regularization.ApplyStep |
ApplyStep determines how the regularization interacts with the optimization process - i.e., when it is applied
relative to updaters like Adam, Nesterov momentum, SGD, etc.
|
Copyright © 2019. All rights reserved.