org.nd4j.parameterserver.distributed.v2.util

Class MessageSplitter

java.lang.Object

org.nd4j.parameterserver.distributed.v2.util.MessageSplitter

public class MessageSplitter
extends java.lang.Object

This class provides methods for splitting VoidMessages into chunks, and merging them back again

Author:

[email protected]

Field Summary

Fields

Modifier and Type	Field and Description
protected java.util.concurrent.atomic.AtomicLong	memoryUse
protected java.util.Map<java.lang.String,ChunksTracker>	trackers

Constructor Summary

Constructors

Constructor and Description
MessageSplitter()

Method Summary

Modifier and Type	Method and Description
static MessageSplitter	getInstance() This method returns shared instance of MessageSplitter
<T extends VoidMessage> org.nd4j.linalg.primitives.Optional<T>	merge(VoidChunk chunk) This method tries to merge using files tracker
<T extends VoidMessage> org.nd4j.linalg.primitives.Optional<T>	merge(VoidChunk chunk, long memoryLimit)
void	release(java.lang.String messageId) This method removes specified messageId from tracking
void	reset()
java.util.Collection<VoidChunk>	split(VoidMessage message, int maxBytes) This method splits VoidMessage into chunks, and returns them as Collection

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

trackers

protected java.util.Map<java.lang.String,ChunksTracker> trackers

memoryUse

protected final java.util.concurrent.atomic.AtomicLong memoryUse

Constructor Detail

MessageSplitter

public MessageSplitter()

Method Detail

getInstance

public static MessageSplitter getInstance()

This method returns shared instance of MessageSplitter

Returns:

split

public java.util.Collection<VoidChunk> split(@NonNull
                                             VoidMessage message,
                                             int maxBytes)
                                      throws java.io.IOException

This method splits VoidMessage into chunks, and returns them as Collection

Parameters:

message -

Returns:

Throws:

java.io.IOException

merge

public <T extends VoidMessage> org.nd4j.linalg.primitives.Optional<T> merge(@NonNull
                                                                            VoidChunk chunk)

This method tries to merge using files tracker

Type Parameters:

T -

Parameters:

chunk -

Returns:

release

public void release(java.lang.String messageId)

This method removes specified messageId from tracking

Parameters:

messageId -

merge

public <T extends VoidMessage> org.nd4j.linalg.primitives.Optional<T> merge(@NonNull
                                                                            VoidChunk chunk,
                                                                            long memoryLimit)

Type Parameters:

T -

Parameters:

chunk -

memoryLimit -

Returns:

reset

public void reset()