trait
Layer extends AnyRef
Type Members
-
abstract
type
Input <: Tape
-
abstract
type
Output <: Tape
Abstract Value Members
-
abstract
def
forward(input: Input): Output
Concrete Value Members
-
-
final
def
!=(arg0: Any): Boolean
-
final
def
##(): Int
-
def
+(other: String): String
-
def
->[B](y: B): (Layer, B)
-
-
final
def
==(arg0: Any): Boolean
-
final
def
asInstanceOf[T0]: T0
-
def
clone(): AnyRef
-
-
-
def
ensuring(cond: Boolean, msg: ⇒ Any): Layer
-
-
-
-
def
finalize(): Unit
-
def
formatted(fmtstr: String): String
-
final
def
getClass(): Class[_]
-
def
hashCode(): Int
-
final
def
isInstanceOf[T0]: Boolean
-
-
final
def
notify(): Unit
-
final
def
notifyAll(): Unit
-
final
def
synchronized[T0](arg0: ⇒ T0): T0
-
def
toString(): String
-
final
def
wait(): Unit
-
final
def
wait(arg0: Long, arg1: Int): Unit
-
final
def
wait(arg0: Long): Unit
-
def
→[B](y: B): (Layer, B)
Shadowed Implicit Value Members
-
val
self: Any
-
val
self: Any
Inherited by implicit conversion any2stringadd from
Layer to StringAdd
Inherited by implicit conversion any2ArrowAssoc from
Layer to ArrowAssoc[Layer]
Inherited by implicit conversion any2Ensuring from
Layer to Ensuring[Layer]
A
Layer
represents a neural network. EachLayer
can be included in as a sub-network of anotherLayer
, forming a more complex neural network. The nesting structure ofLayer
can be used to represent mathematical expression or Coarse-grain neural network structure. When a neural network is written, the most elements in it are placeholders. When network training begins, the data enter into the network.Tree structure of
Layer
The above mathematical expression with equivalent codes can be written, by Symbolic, as:
(1.0 + x) * 2.0.toWeight
.2.0.toWeight
represents a variable, of which the initial value is2
. The value updates during neural network iteration.Both Times and Plus are of case class, therefore,
myLayer
is a tree in nested structure consisted of the case class.Times
andPlus
are placeholders.Weight is a
Layer
containing weight, of which the initial value is2.0
.Identity is a
Layer
with equal input and output, which return the same input back. TheIdentity
here is the placeholder ofInput
.Literal is a
Layer
containing a constant.Iteration
Each training of the network is called as an iteration, including two stages: forward and backward, forming a complete process of https://en.wikipedia.org/wiki/Backpropagation
Forward
When invoking
forward
inLayer.Aux[A,B]
,A
is input type,B
is output type, and bothA
andB
are Tape. Now, the codes are interpreted segment by segment as follows.For example:
When invoking
myLayer.forward(inputData)
,forward
ofTimes
shall be invoked first, of which the pseudo codes are as follows:It is
Plus
atmyLayer.operand1
, andWeight
atmyLayer.operand2
. Therefore,upstream1
andupstream2
are the results offorward
ofoperand1
andoperand2
respectively.In a similar way, the
forward
code ofPlus
is similar toforward
ofTimes
. During the invoking forforward
ofPlus
, operand1 isLiteral
, and operand2 isIdentity
. At this point,forward
ofLiteral
andIdentity
of each are invoked respectively.During the invoking for
forward
ofIdentity
, the same input will be returned. The pseudo code forforward
ofIdentity
is as follows:Therefore,
Input
is thex
in mathematical expression(1.0 + x) * 2.0.toWeight
, and in this way,Input
is propagated to the neural network.The return value
outputTape
ofmyLayer.forward
is inTape
type. Therefore, a tree consisted ofTape
will be generated finally with the structure similar to that ofmyLayer
.Therefore, via layer-by-layer propagation, the same
myLayer.forward(inputTape)
is finally returned byIdentity
and combined into the newly generatedTape
tree.The computation result including
forward
ofoutputTape
can be used foroutputTape
, for example:outputTape.value
is the computation result of mathematical expression(1.0 + x) * 2.0.toWeight
Backward
outputTape.backward
is theoutputTape.backward
ofTimes.Output
, of which the pseudo code is as follows:outputTape.upstream1
andoutputTape.upstream2
are the results offorward
ofoperand1
andoperand2
respectively, which are followed bybackward
ofoutputTape.upstream1
andoutputTape.upstream2
.In a similar way, the
backward
code ofPlus
is similar to
backwardof
Times. During the invoking for
backwardof
Plus,
upstream1and
upstream2are the results of
forwardof
Literaland
Identityrespectively. At this point,
backwardof
upstream1and
upstream2of each are invoked respectively.
Weight
updates duringbackward
, refer to updateDoubleAux & Symbolic API
Layer.Aux[A,B]
represents thatInput
is ofA
type, andOutput
is ofB
type.Tape.Aux[C,D]
represents thatData
is ofC
type, andDelta
is ofD
type.Layer.Aux
andType.Aux
can be combined for use. For example,Layer.Aux[Tape.Aux[A,B]
,Tape.Aux[C,D]]
can be used to represent that the input type of alayer
is aTape
, and the data type of thisTape
isA
,delta
type isB
; the output type of alayer
is aTape
, and the data type of thisTape
isC
,delta
type isD
.Aux is a design pattern which realized type refinement and can be used to limit the range of type parameters.
Generally, we will not handwrite
Aux
type, because we can useSymbolic
to acquire the same effect. For example, when used for symbolic method internal variable and return value:Layer.Aux[Tape.Aux[INDArray, INDArray], Tape.Aux[INDArray, INDArray
andINDArray @Symbolic
are equivalent, so we usually useSymbolic
to replace the writing method ofAux
.Symbolic
Backpropagation
type refinement
aux pattern evolution
aux pattern