org.nd4j.graph.ui

Class LogFileWriter

java.lang.Object

org.nd4j.graph.ui.LogFileWriter

public class LogFileWriter
extends Object

Log file writer - for writing append-only log file for UI etc. File format description: There are 2 blocks in the file 1. The "static information" block - containing 0 or more static info frames, followed by a UIStaticInfoRecord with UIInfoType.START_EVENTS to signify the end of the "static information" section (start of events). 2. The "events" block - containing 0 or more event frames Note that once the UIInfoType.START_EVENTS UIStaticInfoRecord marker has been written, no more static information may be written; before this point, no event records may be written. This allows us to scan only the start of the file to get 'static' information such as the graph structure and hardware information. Data (for both sections) is recorded in "frames".
Each frame is formatted as follows:
[header_length_integer, content_length_integer, header_bytes, content_bytes]
These are as follows: header_length_integer: a signed 32-bit integer representing the number of bytes of the header data (stored in the "header_bytes" section of the frame)
content_length_integer: a signed 32-bit integer representing the number of bytes of the content data (stored in the "content_bytes" section of the frame). May be 0 in some cases (such as when recording UIInfoType.START_EVENTS marker
header_bytes: flat-buffer encoded bytes representing either a UIStaticInfoRecord (in static information block) or a UIEvent (in the event block) In both cases, this header object encodes the typo of data that follows. content_bytes: flat-buffer encoded bytes representing the content, of the type specified in the header


To-do list:
* Implement recording of remaining event types
* Handle loading already existing files
*

Author:

Alex Black

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	LogFileWriter.EventSubtype
static class	LogFileWriter.StaticInfo

Constructor Summary

Constructors

Constructor and Description
LogFileWriter(File file)

Method Summary

Modifier and Type	Method and Description
protected int	encodeGraphStructure(com.google.flatbuffers.FlatBufferBuilder fbb, SameDiff sd)
protected org.nd4j.linalg.primitives.Pair<Integer,com.google.flatbuffers.FlatBufferBuilder>	encodeStaticHeader(byte type) Encode the header as a UIStaticInfoRecord instance for the specific UIEventType
List<org.nd4j.linalg.primitives.Pair<UIEvent,com.google.flatbuffers.Table>>	readEvents() Read all of the events.
List<org.nd4j.linalg.primitives.Pair<UIEvent,com.google.flatbuffers.Table>>	readEvents(long startOffset) Read all of the events starting at a specific file offset
LogFileWriter.StaticInfo	readStatic() Read all static information at the start of the file
boolean	registeredEventName(String name)
long	registerEventName(String name) Register the event name - "accuracy", "loss", etc for later use in recording events.
long	registerEventNameQuiet(String name)
long	writeFinishStaticMarker() Write marker for final static data
long	writeGraphStructure(SameDiff sd) Write the graph structure
long	writeHistogramEventCustomBins(String name, LogFileWriter.EventSubtype subtype, long time, int iteration, int epoch, INDArray bins, INDArray y)
long	writeHistogramEventDiscrete(String name, LogFileWriter.EventSubtype subtype, long time, int iteration, int epoch, List<String> binLabels, INDArray y)
long	writeHistogramEventEqualSpacing(String name, LogFileWriter.EventSubtype subtype, long time, int iteration, int epoch, double min, double max, INDArray y)
long	writeScalarEvent(String name, LogFileWriter.EventSubtype subtype, long time, int iteration, int epoch, Number scalar) Write a single scalar event to the file

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

LogFileWriter

public LogFileWriter(File file)
              throws IOException

Throws:

IOException

Method Detail

writeGraphStructure

public long writeGraphStructure(SameDiff sd)
                         throws IOException

Write the graph structure

Parameters:

sd - SameDiff instance to write the graph structure for

Returns:

Number of bytes written

Throws:

IOException

writeFinishStaticMarker

public long writeFinishStaticMarker()
                             throws IOException

Write marker for final static data

Returns:

Throws:

IOException

readStatic

public LogFileWriter.StaticInfo readStatic()
                                    throws IOException

Read all static information at the start of the file

Returns:

Throws:

IOException

readEvents

public List<org.nd4j.linalg.primitives.Pair<UIEvent,com.google.flatbuffers.Table>> readEvents()
                                                                                       throws IOException

Read all of the events.

Returns:

All of the UI events

Throws:

IOException

readEvents

public List<org.nd4j.linalg.primitives.Pair<UIEvent,com.google.flatbuffers.Table>> readEvents(long startOffset)
                                                                                       throws IOException

Read all of the events starting at a specific file offset

Returns:

All of the UI events

Throws:

IOException

registeredEventName

public boolean registeredEventName(String name)

registerEventNameQuiet

public long registerEventNameQuiet(String name)

registerEventName

public long registerEventName(String name)
                       throws IOException

Register the event name - "accuracy", "loss", etc for later use in recording events.

Parameters:

name - Name to register

Returns:

Number of bytes written

Throws:

IOException

writeScalarEvent

public long writeScalarEvent(String name,
                             LogFileWriter.EventSubtype subtype,
                             long time,
                             int iteration,
                             int epoch,
                             Number scalar)
                      throws IOException

Write a single scalar event to the file

Parameters:

name - Name of the event. Must be registered by registerEventName(String) first!

time - Timestamp

iteration - Iteration for the event

epoch - Epoch for the event

scalar - Scalar value to write

Returns:

Number of bytes written

Throws:

IOException

writeHistogramEventDiscrete

public long writeHistogramEventDiscrete(@NonNull
                                        String name,
                                        LogFileWriter.EventSubtype subtype,
                                        long time,
                                        int iteration,
                                        int epoch,
                                        List<String> binLabels,
                                        @NonNull
                                        INDArray y)
                                 throws IOException

Throws:

IOException

writeHistogramEventEqualSpacing

public long writeHistogramEventEqualSpacing(String name,
                                            LogFileWriter.EventSubtype subtype,
                                            long time,
                                            int iteration,
                                            int epoch,
                                            double min,
                                            double max,
                                            INDArray y)
                                     throws IOException

Throws:

IOException

writeHistogramEventCustomBins

public long writeHistogramEventCustomBins(String name,
                                          LogFileWriter.EventSubtype subtype,
                                          long time,
                                          int iteration,
                                          int epoch,
                                          INDArray bins,
                                          INDArray y)
                                   throws IOException

Throws:

IOException

encodeStaticHeader

protected org.nd4j.linalg.primitives.Pair<Integer,com.google.flatbuffers.FlatBufferBuilder> encodeStaticHeader(byte type)

Encode the header as a UIStaticInfoRecord instance for the specific UIEventType

Parameters:

type - UIEventType

encodeGraphStructure

protected int encodeGraphStructure(com.google.flatbuffers.FlatBufferBuilder fbb,
                                   SameDiff sd)