com.intel.analytics.bigdl.nn

BatchNormalization

class BatchNormalization[T] extends TensorModule[T] with Initializable

This layer implements Batch Normalization as described in the paper: "Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift" by Sergey Ioffe, Christian Szegedy https://arxiv.org/abs/1502.03167

This implementation is useful for inputs NOT coming from convolution layers. For convolution layers, use nn.SpatialBatchNormalization.

The operation implemented is: ( x - mean(x) ) y = -------------------- * gamma + beta standard-deviation(x) where gamma and beta are learnable parameters.The learning of gamma and beta is optional.

T

numeric type

Annotations
@SerialVersionUID( 3181824540272906068L )
Linear Supertypes
Initializable, TensorModule[T], AbstractModule[Tensor[T], Tensor[T], T], Serializable, Serializable, AnyRef, Any
Known Subclasses
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. BatchNormalization
  2. Initializable
  3. TensorModule
  4. AbstractModule
  5. Serializable
  6. Serializable
  7. AnyRef
  8. Any
  1. Hide All
  2. Show all
Learn more about member selection
Visibility
  1. Public
  2. All

Instance Constructors

  1. new BatchNormalization(nOutput: Int, eps: Double = 1.0E-5, momentum: Double = 0.1, affine: Boolean = true, initWeight: Tensor[T] = null, initBias: Tensor[T] = null, initGradWeight: Tensor[T] = null, initGradBias: Tensor[T] = null)(implicit arg0: ClassTag[T], ev: TensorNumeric[T])

    nOutput

    output feature map number

    eps

    avoid divide zero

    momentum

    momentum for weight update

    affine

    affine operation on output or not

    ev

    numeric operator

Value Members

  1. final def !=(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  2. final def !=(arg0: Any): Boolean

    Definition Classes
    Any
  3. final def ##(): Int

    Definition Classes
    AnyRef → Any
  4. final def ==(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  5. final def ==(arg0: Any): Boolean

    Definition Classes
    Any
  6. def accGradParameters(input: Tensor[T], gradOutput: Tensor[T]): Unit

    Computing the gradient of the module with respect to its own parameters.

    Computing the gradient of the module with respect to its own parameters. Many modules do not perform this step as they do not have any parameters. The state variable name for the parameters is module dependent. The module is expected to accumulate the gradients with respect to the parameters in some variable.

    input
    gradOutput

    Definition Classes
    BatchNormalizationAbstractModule
  7. val affine: Boolean

    affine operation on output or not

  8. def apply(name: String): Option[AbstractModule[Activity, Activity, T]]

    Find a module with given name.

    Find a module with given name. If there is no module with given name, it will return None. If there are multiple modules with the given name, an exception will be thrown.

    name
    returns

    Definition Classes
    AbstractModule
  9. final def asInstanceOf[T0]: T0

    Definition Classes
    Any
  10. def backward(input: Tensor[T], gradOutput: Tensor[T], theGradInput: Tensor[T] = null, theGradWeight: Tensor[T] = null, theGradBias: Tensor[T] = null): Tensor[T]

  11. def backward(input: Tensor[T], gradOutput: Tensor[T]): Tensor[T]

    Performs a back-propagation step through the module, with respect to the given input.

    Performs a back-propagation step through the module, with respect to the given input. In general this method makes the assumption forward(input) has been called before, with the same input. This is necessary for optimization reasons. If you do not respect this rule, backward() will compute incorrect gradients.

    input

    input data

    gradOutput

    gradient of next layer

    returns

    gradient corresponding to input data

    Definition Classes
    BatchNormalizationAbstractModule
  12. var backwardTime: Long

    Attributes
    protected
    Definition Classes
    AbstractModule
  13. val bias: Tensor[T]

  14. var biasInitMethod: InitializationMethod

    Attributes
    protected
    Definition Classes
    Initializable
  15. def canEqual(other: Any): Boolean

    Definition Classes
    BatchNormalizationAbstractModule
  16. def checkEngineType(): BatchNormalization.this.type

    get execution engine type

    get execution engine type

    Definition Classes
    AbstractModule
  17. def clearState(): BatchNormalization.this.type

    Clear cached activities to save storage space or network bandwidth.

    Clear cached activities to save storage space or network bandwidth. Note that we use Tensor.set to keep some information like tensor share

    The subclass should override this method if it allocate some extra resource, and call the super.clearState in the override method

    returns

    Definition Classes
    AbstractModule
  18. def clone(): AnyRef

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  19. def cloneModule(): AbstractModule[Tensor[T], Tensor[T], T]

    Definition Classes
    AbstractModule
  20. def copyStatus(src: Module[T]): BatchNormalization.this.type

    Copy the useful running status from src to this.

    Copy the useful running status from src to this.

    The subclass should override this method if it has some parameters besides weight and bias. Such as runningMean and runningVar of BatchNormalization.

    src

    source Module

    returns

    this

    Definition Classes
    BatchNormalizationAbstractModule
  21. val eps: Double

    avoid divide zero

  22. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  23. def equals(other: Any): Boolean

    Definition Classes
    BatchNormalizationAbstractModule → AnyRef → Any
  24. def evaluate(dataSet: LocalDataSet[MiniBatch[T]], vMethods: Array[ValidationMethod[T]]): Array[(ValidationResult, ValidationMethod[T])]

    Definition Classes
    AbstractModule
  25. def evaluate(dataset: RDD[Sample[T]], vMethods: Array[ValidationMethod[T]], batchSize: Option[Int] = None): Array[(ValidationResult, ValidationMethod[T])]

    use ValidationMethod to evaluate module

    use ValidationMethod to evaluate module

    dataset

    dataset for test

    vMethods

    validation methods

    batchSize

    total batchsize of all partitions, optional param and default 4 * partitionNum of dataset

    returns

    Definition Classes
    AbstractModule
  26. def evaluate(): BatchNormalization.this.type

    Definition Classes
    AbstractModule
  27. def finalize(): Unit

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  28. final def forward(input: Tensor[T]): Tensor[T]

    Takes an input object, and computes the corresponding output of the module.

    Takes an input object, and computes the corresponding output of the module. After a forward, the output state variable should have been updated to the new value.

    input

    input data

    returns

    output data

    Definition Classes
    AbstractModule
  29. var forwardTime: Long

    Attributes
    protected
    Definition Classes
    AbstractModule
  30. final def getClass(): Class[_]

    Definition Classes
    AnyRef → Any
  31. def getName(): String

    Get the module name, default name is className@namePostfix

    Get the module name, default name is className@namePostfix

    returns

    Definition Classes
    AbstractModule
  32. def getNumericType(): TensorDataType

    returns

    Float or Double

    Definition Classes
    AbstractModule
  33. def getParameters(): (Tensor[T], Tensor[T])

    This method compact all parameters and gradients of the model into two tensors.

    This method compact all parameters and gradients of the model into two tensors. So it's easier to use optim method

    returns

    Definition Classes
    AbstractModule
  34. def getParametersTable(): Table

    This function returns a table contains ModuleName, the parameter names and parameter value in this module.

    This function returns a table contains ModuleName, the parameter names and parameter value in this module. The result table is a structure of Table(ModuleName -> Table(ParameterName -> ParameterValue)), and the type is Table[String, Table[String, Tensor[T]]].

    For example, get the weight of a module named conv1: table[Table]("conv1")[Tensor[T]]("weight").

    Custom modules should override this function if they have parameters.

    returns

    Table

    Definition Classes
    BatchNormalizationAbstractModule
  35. def getPrintName(): String

    Attributes
    protected
    Definition Classes
    AbstractModule
  36. def getScaleB(): Double

    Get the scale of gradientBias

    Get the scale of gradientBias

    Definition Classes
    AbstractModule
  37. def getScaleW(): Double

    Get the scale of gradientWeight

    Get the scale of gradientWeight

    Definition Classes
    AbstractModule
  38. def getTimes(): Array[(AbstractModule[_ <: Activity, _ <: Activity, T], Long, Long)]

    Definition Classes
    AbstractModule
  39. def getWeightsBias(): Array[Tensor[T]]

    Get weight and bias for the module

    Get weight and bias for the module

    returns

    array of weights and bias

    Definition Classes
    AbstractModule
  40. val gradBias: Tensor[T]

  41. var gradInput: Tensor[T]

    The cached gradient of activities.

    The cached gradient of activities. So we don't compute it again when need it

    Definition Classes
    AbstractModule
  42. val gradWeight: Tensor[T]

  43. def hashCode(): Int

    Definition Classes
    BatchNormalizationAbstractModule → AnyRef → Any
  44. def inputs(nodes: ModuleNode[T]*): ModuleNode[T]

    Some other modules point to current module

    Some other modules point to current module

    nodes

    upstream module nodes

    returns

    node containing current module

    Definition Classes
    AbstractModule
  45. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any
  46. final def isTraining(): Boolean

    Definition Classes
    AbstractModule
  47. var line: String

    Attributes
    protected
    Definition Classes
    AbstractModule
  48. def loadModelWeights(srcModel: Module[Float], matchAll: Boolean = true): BatchNormalization.this.type

    copy weights from another model, mapping by layer name

    copy weights from another model, mapping by layer name

    srcModel

    model to copy from

    matchAll

    whether to match all layers' weights and bias,

    returns

    current module

    Definition Classes
    AbstractModule
  49. def loadWeights(weightPath: String, matchAll: Boolean = true): BatchNormalization.this.type

    load pretrained weights and bias to current module

    load pretrained weights and bias to current module

    weightPath

    file to store weights and bias

    matchAll

    whether to match all layers' weights and bias, if not, only load existing pretrained weights and bias

    returns

    current module

    Definition Classes
    AbstractModule
  50. val momentum: Double

    momentum for weight update

  51. val nDim: Int

  52. val nOutput: Int

    output feature map number

  53. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  54. final def notify(): Unit

    Definition Classes
    AnyRef
  55. final def notifyAll(): Unit

    Definition Classes
    AnyRef
  56. var output: Tensor[T]

    The cached output.

    The cached output. So we don't compute it again when need it

    Definition Classes
    AbstractModule
  57. def parameters(): (Array[Tensor[T]], Array[Tensor[T]])

    This function returns two arrays.

    This function returns two arrays. One for the weights and the other the gradients Custom modules should override this function if they have parameters

    returns

    (Array of weights, Array of grad)

    Definition Classes
    BatchNormalizationAbstractModule
  58. def predict(dataset: RDD[Sample[T]]): RDD[Activity]

    module predict, return the probability distribution

    module predict, return the probability distribution

    dataset

    dataset for prediction

    Definition Classes
    AbstractModule
  59. def predictClass(dataset: RDD[Sample[T]]): RDD[Int]

    module predict, return the predict label

    module predict, return the predict label

    dataset

    dataset for prediction

    Definition Classes
    AbstractModule
  60. def reset(): Unit

  61. def resetTimes(): Unit

    Definition Classes
    AbstractModule
  62. val runningMean: Tensor[T]

  63. val runningVar: Tensor[T]

  64. def save(path: String, overWrite: Boolean = false): BatchNormalization.this.type

    Save this module to path.

    Save this module to path.

    path

    path to save module, local file system, HDFS and Amazon S3 is supported. HDFS path should be like "hdfs://[host]:[port]/xxx" Amazon S3 path should be like "s3a://bucket/xxx"

    overWrite

    if overwrite

    returns

    self

    Definition Classes
    AbstractModule
  65. def saveCaffe(prototxtPath: String, modelPath: String, useV2: Boolean = true, overwrite: Boolean = false): BatchNormalization.this.type

    Definition Classes
    AbstractModule
  66. val saveMean: Tensor[T]

  67. val saveStd: Tensor[T]

  68. def saveTF(inputs: Seq[(String, Seq[Int])], path: String, byteOrder: ByteOrder = ByteOrder.LITTLE_ENDIAN, dataFormat: TensorflowDataFormat = TensorflowDataFormat.NHWC): BatchNormalization.this.type

    Definition Classes
    AbstractModule
  69. def saveTorch(path: String, overWrite: Boolean = false): BatchNormalization.this.type

    Definition Classes
    AbstractModule
  70. def saveWeights(path: String, overWrite: Boolean): Unit

    save weights and bias to file

    save weights and bias to file

    path

    file to save

    overWrite

    whether to overwrite or not

    Definition Classes
    AbstractModule
  71. var scaleB: Double

    Attributes
    protected
    Definition Classes
    AbstractModule
  72. var scaleW: Double

    The scale of gradient weight and gradient bias before gradParameters being accumulated.

    The scale of gradient weight and gradient bias before gradParameters being accumulated.

    Attributes
    protected
    Definition Classes
    AbstractModule
  73. def setInit(status: Boolean = true): BatchNormalization.this.type

    Annotations
    @inline()
  74. def setInitMethod(weightInitMethod: InitializationMethod = null, biasInitMethod: InitializationMethod = null): BatchNormalization.this.type

    Definition Classes
    Initializable
  75. def setLine(line: String): BatchNormalization.this.type

    Definition Classes
    AbstractModule
  76. def setName(name: String): BatchNormalization.this.type

    Set the module name

    Set the module name

    name
    returns

    Definition Classes
    AbstractModule
  77. def setScaleB(b: Double): BatchNormalization.this.type

    Set the scale of gradientBias

    Set the scale of gradientBias

    b

    the value of the scale of gradientBias

    returns

    this

    Definition Classes
    AbstractModule
  78. def setScaleW(w: Double): BatchNormalization.this.type

    Set the scale of gradientWeight

    Set the scale of gradientWeight

    w

    the value of the scale of gradientWeight

    returns

    this

    Definition Classes
    AbstractModule
  79. def setWeightsBias(newWeights: Array[Tensor[T]]): BatchNormalization.this.type

    Set weight and bias for the module

    Set weight and bias for the module

    newWeights

    array of weights and bias

    returns

    Definition Classes
    AbstractModule
  80. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef
  81. def toString(): String

    Definition Classes
    BatchNormalizationAbstractModule → AnyRef → Any
  82. var train: Boolean

    Module status.

    Module status. It is useful for modules like dropout/batch normalization

    Attributes
    protected
    Definition Classes
    AbstractModule
  83. def training(): BatchNormalization.this.type

    Definition Classes
    AbstractModule
  84. def updateGradInput(input: Tensor[T], gradOutput: Tensor[T]): Tensor[T]

    Computing the gradient of the module with respect to its own input.

    Computing the gradient of the module with respect to its own input. This is returned in gradInput. Also, the gradInput state variable is updated accordingly.

    input
    gradOutput
    returns

    Definition Classes
    BatchNormalizationAbstractModule
  85. def updateOutput(input: Tensor[T]): Tensor[T]

    Computes the output using the current parameter set of the class and input.

    Computes the output using the current parameter set of the class and input. This function returns the result which is stored in the output field.

    input
    returns

    Definition Classes
    BatchNormalizationAbstractModule
  86. def updateParameters(learningRate: T): Unit

    Definition Classes
    AbstractModule
  87. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  88. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  89. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  90. val weight: Tensor[T]

  91. var weightInitMethod: InitializationMethod

    Attributes
    protected
    Definition Classes
    Initializable
  92. def zeroGradParameters(): Unit

    If the module has parameters, this will zero the accumulation of the gradients with respect to these parameters.

    If the module has parameters, this will zero the accumulation of the gradients with respect to these parameters. Otherwise, it does nothing.

    Definition Classes
    BatchNormalizationAbstractModule

Inherited from Initializable

Inherited from TensorModule[T]

Inherited from AbstractModule[Tensor[T], Tensor[T], T]

Inherited from Serializable

Inherited from Serializable

Inherited from AnyRef

Inherited from Any

Ungrouped