org.deeplearning4j.nn.conf.layers

Class SelfAttentionLayer

java.lang.Object

org.deeplearning4j.nn.conf.layers.Layer

org.deeplearning4j.nn.conf.layers.samediff.AbstractSameDiffLayer

org.deeplearning4j.nn.conf.layers.samediff.SameDiffLayer

org.deeplearning4j.nn.conf.layers.SelfAttentionLayer

All Implemented Interfaces:

Serializable, Cloneable, TrainingConfig

public class SelfAttentionLayer
extends SameDiffLayer

Implements Dot Product Self Attention Takes in RNN style input in the shape of [batchSize, features, timesteps] and applies dot product attention using each timestep as the query. The output will be in the shape of [batchSize, nOut, timesteps]. Attention implemented as in Attention is all you need by Vaswani et al. [arXiv:1706.03762], pp. 4,5

Author:

Paul Dubs

See Also:

LearnedSelfAttentionLayer, RecurrentAttentionLayer, MultiHeadDotProductAttention, Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	SelfAttentionLayer.Builder

Field Summary

Fields inherited from class org.deeplearning4j.nn.conf.layers.samediff.SameDiffLayer

paramWeightInit, weightInit

Fields inherited from class org.deeplearning4j.nn.conf.layers.samediff.AbstractSameDiffLayer

biasUpdater, gradientNormalization, gradientNormalizationThreshold, regularization, regularizationBias, updater

Fields inherited from class org.deeplearning4j.nn.conf.layers.Layer

constraints, iDropout, layerName

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	SelfAttentionLayer(SelfAttentionLayer.Builder builder)

Method Summary

Modifier and Type	Method and Description
SDVariable	defineLayer(SameDiff sameDiff, SDVariable layerInput, Map<String,SDVariable> paramTable, SDVariable mask) Define the layer
void	defineParameters(SDLayerParams params) Define the parameters for the network.
InputType	getOutputType(int layerIndex, InputType inputType) For a given type of input to this layer, what is the type of the output?
InputPreProcessor	getPreProcessorForInputType(InputType inputType) For the given type of input to this layer, what preprocessor (if any) is required? Returns null if no preprocessor is required, otherwise returns an appropriate InputPreProcessor for this layer, such as a CnnToFeedForwardPreProcessor
void	initializeParameters(Map<String,INDArray> params) Set the initial parameter values for this layer, if required
void	setNIn(InputType inputType, boolean override) Set the nIn value (number of inputs, or input channels for CNNs) based on the given input type

Methods inherited from class org.deeplearning4j.nn.conf.layers.samediff.SameDiffLayer

feedForwardMaskArray, instantiate, validateInput

Methods inherited from class org.deeplearning4j.nn.conf.layers.samediff.AbstractSameDiffLayer

applyGlobalConfig, applyGlobalConfigToLayer, getLayerParams, getMemoryReport, getRegularizationByParam, getUpdaterByParam, initializer, initWeights, isPretrainParam, onesMaskForInput, paramReshapeOrder

Methods inherited from class org.deeplearning4j.nn.conf.layers.Layer

clone, initializeConstraints, resetLayerDefaultConfig, setDataType

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.deeplearning4j.nn.api.TrainingConfig

getGradientNormalization, getGradientNormalizationThreshold, getLayerName

Constructor Detail

SelfAttentionLayer

protected SelfAttentionLayer(SelfAttentionLayer.Builder builder)

Method Detail

getPreProcessorForInputType

public InputPreProcessor getPreProcessorForInputType(InputType inputType)

Description copied from class: Layer

For the given type of input to this layer, what preprocessor (if any) is required?
Returns null if no preprocessor is required, otherwise returns an appropriate InputPreProcessor for this layer, such as a CnnToFeedForwardPreProcessor

Overrides:

getPreProcessorForInputType in class AbstractSameDiffLayer

Parameters:

inputType - InputType to this layer

Returns:

Null if no preprocessor is required, otherwise the type of preprocessor necessary for this layer/input combination

setNIn

public void setNIn(InputType inputType,
                   boolean override)

Description copied from class: Layer

Set the nIn value (number of inputs, or input channels for CNNs) based on the given input type

Overrides:

setNIn in class AbstractSameDiffLayer

Parameters:

inputType - Input type for this layer

override - If false: only set the nIn value if it's not already set. If true: set it regardless of whether it's already set or not.

getOutputType

public InputType getOutputType(int layerIndex,
                               InputType inputType)

Description copied from class: Layer

For a given type of input to this layer, what is the type of the output?

Specified by:

getOutputType in class Layer

Parameters:

layerIndex - Index of the layer

inputType - Type of input for the layer

Returns:

Type of output from the layer

defineParameters

public void defineParameters(SDLayerParams params)

Description copied from class: AbstractSameDiffLayer

Define the parameters for the network. Use SDLayerParams.addWeightParam(String, long...) and SDLayerParams.addBiasParam(String, long...)

Specified by:

defineParameters in class AbstractSameDiffLayer

Parameters:

params - Object used to set parameters for this layer

initializeParameters

public void initializeParameters(Map<String,INDArray> params)

Description copied from class: AbstractSameDiffLayer

Set the initial parameter values for this layer, if required

Specified by:

initializeParameters in class AbstractSameDiffLayer

Parameters:

params - Parameter arrays that may be initialized

defineLayer

public SDVariable defineLayer(SameDiff sameDiff,
                              SDVariable layerInput,
                              Map<String,SDVariable> paramTable,
                              SDVariable mask)

Description copied from class: SameDiffLayer

Define the layer

Specified by:

defineLayer in class SameDiffLayer

Parameters:

sameDiff - SameDiff instance

layerInput - Input to the layer

paramTable - Parameter table - keys as defined by AbstractSameDiffLayer.defineParameters(SDLayerParams)

mask - Optional, maybe null. Mask to apply if supported

Returns:

The final layer variable corresponding to the activations/output from the forward pass