Package org.deeplearning4j.optimize.solvers.accumulation

Class EncodedGradientsAccumulator

java.lang.Object

org.deeplearning4j.optimize.solvers.accumulation.EncodedGradientsAccumulator

All Implemented Interfaces:

Serializable, GradientsAccumulator, Registerable

public class EncodedGradientsAccumulator
extends Object
implements GradientsAccumulator, Registerable

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	EncodedGradientsAccumulator.Builder

Field Summary

Fields

Modifier and Type	Field	Description
protected ThreadLocal<INDArray>	accumulator
protected WorkspaceConfiguration	appliedConfiguration
protected AtomicInteger	barrier
protected Integer	boundary
protected AtomicBoolean	bypassMode
protected AtomicInteger	currentConsumers
static long	DEFAULT_INITIAL_MEMORY
protected boolean	encodingDebugMode
protected IndexedTail	externalSource
protected AtomicBoolean	externalUpdatesAvailable
protected MessageHandler	handler
protected ThreadLocal<Integer>	index
protected long	initialMemory
protected boolean	isDebug
protected AtomicBoolean	isDone
protected AtomicBoolean	isFirst
protected List<ReentrantLock>	locks
protected List<BlockingQueue<INDArray>>	messages
protected int	parties
protected int	queueSize
protected AtomicBoolean	registered
protected boolean	relocatable
protected AtomicInteger	secondary
protected AtomicThrowable	throwable
protected ThreadLocal<AtomicLong>	updatesApplied
protected AtomicInteger	workersCounter
protected List<MemoryWorkspace>	workspaces

Constructor Summary

Constructors

Constructor	Description
EncodedGradientsAccumulator(int parties, double threshold)
EncodedGradientsAccumulator(int parties, @NonNull MessageHandler handler, long initialMemory, int queueSize, Integer boundary, boolean encodingDebugMode)
EncodedGradientsAccumulator(int parties, ThresholdAlgorithm thresholdAlgorithm, ResidualPostProcessor residualPostProcessor, boolean encodingDebugMode)

Method Summary

Modifier and Type	Method	Description
void	applyUpdate(StepFunction function, INDArray params, INDArray updates, boolean isFinalStep)	This method applies accumulated updates via given StepFunction
void	applyUpdate(StepFunction function, INDArray params, INDArray updates, double alpha)	This method applies accumulated updates via given StepFunction
void	fallbackToSingleConsumerMode(boolean reallyFallback)	This method enables/disables bypass mode
IndexedTail	getExternalSource()
static long	getOptimalBufferSize(long paramsLength, int numWorkers, int queueSize)	This method returns optimal bufferSize for a given model We know, that updates are guaranteed to have MAX size of params / 16.
static long	getOptimalBufferSize(Model model, int numWorkers, int queueSize)
boolean	hasAnything()	This method checks if there are any (probably external) updates available
void	markExternalUpdates(boolean updatesAvailable)	This method allows to highlight early availability of updates
void	receiveUpdate(INDArray array)	This method accepts updates suitable for StepFunction and puts them to the queue, which is used in backpropagation loop
void	registerConsumers(int numConsumers)	This method notifies producer about number of consumers for the current consumption cycle
void	reset()	This method resets all accumulated updates (if any)
void	setExternalSource(IndexedTail source)	This method allows to pass external updates to accumulator, they will be populated across all workers using this GradientsAccumulator instance
void	storeUpdate(INDArray array, int iterationNumber, int epochNumber)	This method accepts updates suitable for StepFunction, and accumulates/propagates it across all workers
protected void	synchronize(int consumers)
protected void	synchronize(int consumers, boolean finalLock)
void	touch()	This method does initialization of given worker wrt Thread-Device Affinity

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DEFAULT_INITIAL_MEMORY

public static final long DEFAULT_INITIAL_MEMORY

See Also:

Constant Field Values

accumulator

protected ThreadLocal<INDArray> accumulator

parties

protected int parties

handler

protected MessageHandler handler

messages

protected List<BlockingQueue<INDArray>> messages

workspaces

protected List<MemoryWorkspace> workspaces

locks

protected List<ReentrantLock> locks

workersCounter

protected AtomicInteger workersCounter

index

protected ThreadLocal<Integer> index

initialMemory

protected long initialMemory

queueSize

protected int queueSize

boundary

protected Integer boundary

encodingDebugMode

protected boolean encodingDebugMode

externalSource

protected IndexedTail externalSource

isFirst

protected AtomicBoolean isFirst

isDone

protected AtomicBoolean isDone

barrier

protected AtomicInteger barrier

secondary

protected AtomicInteger secondary

registered

protected AtomicBoolean registered

bypassMode

protected AtomicBoolean bypassMode

currentConsumers

protected final AtomicInteger currentConsumers

throwable

protected final AtomicThrowable throwable

isDebug

protected boolean isDebug

relocatable

protected final boolean relocatable

updatesApplied

protected ThreadLocal<AtomicLong> updatesApplied

externalUpdatesAvailable

protected AtomicBoolean externalUpdatesAvailable

appliedConfiguration

protected WorkspaceConfiguration appliedConfiguration

Constructor Detail

EncodedGradientsAccumulator

public EncodedGradientsAccumulator(int parties,
                                   double threshold)

EncodedGradientsAccumulator

public EncodedGradientsAccumulator(int parties,
                                   ThresholdAlgorithm thresholdAlgorithm,
                                   ResidualPostProcessor residualPostProcessor,
                                   boolean encodingDebugMode)

EncodedGradientsAccumulator

public EncodedGradientsAccumulator(int parties,
                                   @NonNull
                                   @NonNull MessageHandler handler,
                                   long initialMemory,
                                   int queueSize,
                                   Integer boundary,
                                   boolean encodingDebugMode)

Method Detail

getOptimalBufferSize

public static long getOptimalBufferSize(long paramsLength,
                                        int numWorkers,
                                        int queueSize)

This method returns optimal bufferSize for a given model We know, that updates are guaranteed to have MAX size of params / 16. So, here we go. I.e. for model with 100m params, that's 400m of floats (or 800m of doubles) The worst case for us is bitmap encoding, that takes 2 bits to encode each gradient value so, for float in worst case we'll have (100m / 16) int elements. So, our buffer size will be 6.25m * queueSize * 4 bytes per int

Parameters:

paramsLength -

numWorkers -

queueSize -

Returns:

getOptimalBufferSize

public static long getOptimalBufferSize(Model model,
                                        int numWorkers,
                                        int queueSize)

fallbackToSingleConsumerMode

public void fallbackToSingleConsumerMode(boolean reallyFallback)

Description copied from interface: Registerable

This method enables/disables bypass mode

Specified by:

fallbackToSingleConsumerMode in interface Registerable

registerConsumers

public void registerConsumers(int numConsumers)

Description copied from interface: Registerable

This method notifies producer about number of consumers for the current consumption cycle

Specified by:

registerConsumers in interface Registerable

getExternalSource

public IndexedTail getExternalSource()

Specified by:

getExternalSource in interface GradientsAccumulator

markExternalUpdates

public void markExternalUpdates(boolean updatesAvailable)

Description copied from interface: GradientsAccumulator

This method allows to highlight early availability of updates

Specified by:

markExternalUpdates in interface GradientsAccumulator

synchronize

protected void synchronize(int consumers)

synchronize

protected void synchronize(int consumers,
                           boolean finalLock)

applyUpdate

public void applyUpdate(StepFunction function,
                        INDArray params,
                        INDArray updates,
                        boolean isFinalStep)

This method applies accumulated updates via given StepFunction

Specified by:

applyUpdate in interface GradientsAccumulator

Parameters:

function -

params -

applyUpdate

public void applyUpdate(StepFunction function,
                        INDArray params,
                        INDArray updates,
                        double alpha)

This method applies accumulated updates via given StepFunction

Specified by:

applyUpdate in interface GradientsAccumulator

Parameters:

function -

params -

alpha -

setExternalSource

public void setExternalSource(IndexedTail source)

This method allows to pass external updates to accumulator, they will be populated across all workers using this GradientsAccumulator instance

Specified by:

setExternalSource in interface GradientsAccumulator

Parameters:

source -

touch

public void touch()

This method does initialization of given worker wrt Thread-Device Affinity

Specified by:

touch in interface GradientsAccumulator

storeUpdate

public void storeUpdate(INDArray array,
                        int iterationNumber,
                        int epochNumber)

This method accepts updates suitable for StepFunction, and accumulates/propagates it across all workers

Specified by:

storeUpdate in interface GradientsAccumulator

Parameters:

array -

receiveUpdate

public void receiveUpdate(INDArray array)

This method accepts updates suitable for StepFunction and puts them to the queue, which is used in backpropagation loop

PLEASE NOTE: array is expected to be ready for use and match params dimensionality

Specified by:

receiveUpdate in interface GradientsAccumulator

Parameters:

array -

reset

public void reset()

This method resets all accumulated updates (if any)

Specified by:

reset in interface GradientsAccumulator

hasAnything

public boolean hasAnything()

Description copied from interface: GradientsAccumulator

This method checks if there are any (probably external) updates available

Specified by:

hasAnything in interface GradientsAccumulator

Returns: