org.nd4j.autodiff.samediff.internal.memory

Class ArrayCacheMemoryMgr

java.lang.Object

org.nd4j.autodiff.samediff.internal.memory.AbstractMemoryMgr

org.nd4j.autodiff.samediff.internal.memory.ArrayCacheMemoryMgr

All Implemented Interfaces:

Closeable, AutoCloseable, SessionMemMgr

public class ArrayCacheMemoryMgr
extends AbstractMemoryMgr

ArrayCacheMemoryMgr reuses arrays to reduce the number of memory allocations and deallocations.
Memory allocations and deallocations can be quite expensive, especially on GPUs.
Note that when arrays are reused, they are reused for the same datatype only.
If caching a released array would result in the the maximum cache size being is exceeded, the oldest arrays will be deallocated first, until the new array can in the cache.

By default, the following parameters are used for the cache:

• Maximum cache size: 0.25 x max memory, where:
• CPU: max memory is determined using Pointer.maxBytes()
• GPU: max memory is determined using GPU 0 total memory
• Larger array max multiple: 2.0
• This means: if an exact array size can't be provided from the cache, use the next smallest array with a buffer up to 2.0x larger than requested
• If no cached arrays of size < 2x requested exists, allocate a new array
• Small array threshold: 1024 elements
• This means: the "larger array max multiple" doesn't apply below this level. For example, we might return a size 1 array backed by a size 1023 buffer

Author:

Alex Black

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
class	ArrayCacheMemoryMgr.ArrayStore

Constructor Summary

Constructors

Constructor and Description
ArrayCacheMemoryMgr() Create an ArrayCacheMemoryMgr with default settings as per ArrayCacheMemoryMgr
ArrayCacheMemoryMgr(double maxMemFrac, long smallArrayThreshold, double largerArrayMaxMultiple)

Method Summary

Modifier and Type	Method and Description
INDArray	allocate(boolean detached, DataType dataType, long... shape) Allocate an array with the specified datatype and shape. NOTE: This array should be assumed to be uninitialized - i.e., contains random values.
INDArray	allocate(boolean detached, LongShapeDescriptor descriptor) As per SessionMemMgr.allocate(boolean, DataType, long...) but from a LongShapeDescriptor instead
void	close() Close the session memory manager and clean up any memory / resources, if any
void	release(@NonNull INDArray array) Release the array.

Methods inherited from class org.nd4j.autodiff.samediff.internal.memory.AbstractMemoryMgr

dup, ulike

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ArrayCacheMemoryMgr

public ArrayCacheMemoryMgr()

Create an ArrayCacheMemoryMgr with default settings as per ArrayCacheMemoryMgr

ArrayCacheMemoryMgr

public ArrayCacheMemoryMgr(double maxMemFrac,
                           long smallArrayThreshold,
                           double largerArrayMaxMultiple)

Parameters:

maxMemFrac - Maximum memory fraciton to use as cache

smallArrayThreshold - Below this size (elements), don't apply the "largerArrayMaxMultiple" rule

largerArrayMaxMultiple - Maximum multiple of the requested size to return from the cache. If an array of size 1024 is requested, and largerArrayMaxMultiple is 2.0, then we'll return from the cache the array with the smallest data buffer up to 2.0*1024 elements; otherwise we'll return a new array

Method Detail

allocate

public INDArray allocate(boolean detached,
                         DataType dataType,
                         long... shape)

Description copied from interface: SessionMemMgr

Allocate an array with the specified datatype and shape.
NOTE: This array should be assumed to be uninitialized - i.e., contains random values.

Parameters:

detached - If true: the array is safe to return outside of the SameDiff session run (for example, the array is one that may be returned to the user)

dataType - Datatype of the returned array

shape - Array shape

Returns:

The newly allocated (uninitialized) array

allocate

public INDArray allocate(boolean detached,
                         LongShapeDescriptor descriptor)

Description copied from interface: SessionMemMgr

As per SessionMemMgr.allocate(boolean, DataType, long...) but from a LongShapeDescriptor instead

release

public void release(@NonNull
                    @NonNull INDArray array)

Description copied from interface: SessionMemMgr

Release the array. All arrays allocated via one of the allocate methods should be returned here once they are no longer used, and all references to them should be cleared. After calling release, anything could occur to the array - deallocated, workspace closed, reused, etc.

Parameters:

array - The array that can be released

close

public void close()

Description copied from interface: SessionMemMgr

Close the session memory manager and clean up any memory / resources, if any