SGD

Type Members

1. case class Default() extends LearningRateSchedule with Product with Serializable

   

   It is the default learning rate schedule.

   It is the default learning rate schedule. For each iteration, the learning rate would update with the following formula:

   l_{n + 1} = l / (1 + n * learning_rate_decay)

   where l is the initial learning rate

2. case class EpochDecay(decayType: (Int) ⇒ Double) extends LearningRateSchedule with Product with Serializable

   

   It is an epoch decay learning rate schedule The learning rate decays through a function argument on number of run epochs

   It is an epoch decay learning rate schedule The learning rate decays through a function argument on number of run epochs

   l_{n + 1} = l_{n} * 0.1 ^ decayType(epoch)

   decayType

   is a function with number of run epochs as the argument

3. case class EpochSchedule(regimes: Array[Regime]) extends LearningRateSchedule with Product with Serializable

   

   EpochSchedule is a learning rate schedule which configure the learning rate according to some pre-defined Regime.

   EpochSchedule is a learning rate schedule which configure the learning rate according to some pre-defined Regime. If the running epoch is within the interval of a regime r [r.startEpoch, r.endEpoch], then the learning rate will take the "learningRate" in r.config.

   regimes

   an array of pre-defined Regime.

4. case class EpochStep(stepSize: Int, gamma: Double) extends LearningRateSchedule with Product with Serializable

   

   EpochStep is a learning rate schedule, which rescale the learning rate by gamma for each stepSize epochs.

   EpochStep is a learning rate schedule, which rescale the learning rate by gamma for each stepSize epochs.

   stepSize

   For how many epochs to update the learning rate once

   gamma

   the rescale factor

5. trait LearningRateSchedule extends AnyRef

   

   Learning rate schedule for SGD

6. case class Poly(power: Double, maxIteration: Int) extends LearningRateSchedule with Product with Serializable

   

   A learning rate decay policy, where the effective learning rate follows a polynomial decay, to be zero by the max_iteration.

   A learning rate decay policy, where the effective learning rate follows a polynomial decay, to be zero by the max_iteration. Calculation: base_lr (1 - iter/maxIteration) ^ (power)

   power

   coeffient of decay, refer to calculation formula

   maxIteration

   max iteration when lr becomes zero

7. case class Regime(startEpoch: Int, endEpoch: Int, config: Table) extends Product with Serializable

   

   A structure to specify hyper parameters by start epoch and end epoch.

   A structure to specify hyper parameters by start epoch and end epoch. Usually work with EpochSchedule.

   startEpoch

   start epoch

   endEpoch

   end epoch

   config

   config table contains hyper parameters

8. case class Step(stepSize: Int, gamma: Double) extends LearningRateSchedule with Product with Serializable

   

   A learning rate decay policy, where the effective learning rate is calculated as base_lr * gamma ^ (floor(iter / stepSize))

   A learning rate decay policy, where the effective learning rate is calculated as base_lr * gamma ^ (floor(iter / stepSize))

   stepSize

   the inteval for lr decay

   gamma

   coefficient of decay, refer to calculation formula