net.lecousin.framework.io.buffering

Class BufferedIO

java.lang.Object

net.lecousin.framework.util.ConcurrentCloseable

net.lecousin.framework.io.buffering.BufferedIO

All Implemented Interfaces:

AutoCloseable, IO, IO.KnownSize, IO.PositionKnown, IO.Readable, IO.Readable.Buffered, IO.Readable.Seekable, IO.ReadableByteStream, IO.Seekable, AsyncCloseable<Exception>, CloseableListenable, IConcurrentCloseable

Direct Known Subclasses:

BufferedIO.ReadWrite

public class BufferedIO
extends ConcurrentCloseable
implements IO.Readable.Seekable, IO.Readable.Buffered, IO.KnownSize

BufferedIO is often the best buffering implementation when we need both bufferization and seek operations.
Basically, it keeps a list of buffers corresponding to the different parts of the IO. On first usage, a part of the IO is loaded into memory, and so read and write operations can be done quickly. After some time without being used, a buffer can be unloaded to free memory.
For a BufferedIO.ReadWrite, write operations are delayed: the data is kept in memory, so can be modified again or read quickly. After 30 seconds without new write operation on a buffer, or if too much data has been written in memory and not yet flushed on the underlying IO, a buffer is automatically flushed.

BufferedIOs are managed by the BufferedIO.MemoryManagement class. It is responsible for the background tasks: write operations and free buffer no longer used.
All BufferedIOs are managed together, so the MemoryManagement can decide of operations based on memory usage considering all instances of BufferedIO.
It checks regularly what is used and not used and which write operations need to be done. In case a maximum memory is reached, and new memory is needed, it will free memory first. But it will try to do operations in background as much as possible, and will decide to free memory when it detects we are close to reach the maximum memory usage.
Using a single MemoryManagement for all BufferedIOs allows to manage memory globally for the application, so many BufferedIOs can be used together while controlling the memory usage, and giving priority to buffers recently used.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	BufferedIO.MemoryManagement Manages memory usage and delayed write operations of BufferedIO instances.
static class	BufferedIO.ReadWrite Writable BufferedIO.

Nested classes/interfaces inherited from interface net.lecousin.framework.io.IO.Seekable

IO.Seekable.SeekType

Nested classes/interfaces inherited from interface net.lecousin.framework.io.IO

IO.KnownSize, IO.OperationType, IO.OutputToInput, IO.PositionKnown, IO.Readable, IO.ReadableByteStream, IO.Resizable, IO.Writable, IO.WritableByteStream, IO.WriterAsync

Nested classes/interfaces inherited from interface net.lecousin.framework.util.CloseableListenable

CloseableListenable.Impl

Nested classes/interfaces inherited from interface net.lecousin.framework.io.IO.Readable

IO.Readable.Buffered, IO.Readable.Seekable

Field Summary

Fields

Modifier and Type	Field and Description
protected int	bufferSize
protected net.lecousin.framework.io.buffering.BufferedIO.BufferTable	bufferTable
protected boolean	closing
protected int	firstBufferSize
protected IO.Readable.Seekable	io
protected BufferedIO.MemoryManagement	memory
protected long	position
protected boolean	preLoadNextBuffer
protected long	size

Constructor Summary

Constructors

Constructor and Description
BufferedIO(IO.Readable.Seekable io, int firstBufferSize, int nextBuffersSize, boolean preLoadNextBuffer) Constructor.
BufferedIO(IO.Readable.Seekable io, long size, int firstBufferSize, int nextBuffersSize, boolean preLoadNextBuffer) Constructor.

Method Summary

Modifier and Type	Method and Description
ISynchronizationPoint<IOException>	canStartReading() Return a synchronization point that is unblocked when data is ready to be read.
protected void	closeResources(SynchronizationPoint<Exception> ondone)
protected ISynchronizationPoint<Exception>	closeUnderlyingResources()
protected long	getBufferIndex(long pos)
protected int	getBufferOffset(long pos)
protected long	getBufferPosition(long index)
long	getPosition() Return the current position.
byte	getPriority() Return the priority.
AsyncWork<Long,IOException>	getSizeAsync() Asynchronous operation to get the size of the IO.
long	getSizeSync() Synchronous operation to get the size of the IO.
String	getSourceDescription() Describe the IO, which can be used for logging or debugging purposes.
TaskManager	getTaskManager() Return the TaskManager used for asynchronous operations.
IO	getWrappedIO() If this IO is wrapping another one (such as adding Buffered capabilities), this method returns it, or null if it does not wrap another IO.
protected void	preLoadBuffer(long pos)
int	read() Read next byte.
int	read(byte[] buffer, int offset, int len) Read bytes into the given buffer, starting to store them at the given offset, and with the given maximum number of bytes.
int	readAsync() Read a single byte if possible.
AsyncWork<Integer,IOException>	readAsync(ByteBuffer buffer, RunnableWithParameter<Pair<Integer,IOException>> ondone) Asynchronous read operation.
AsyncWork<Integer,IOException>	readAsync(long pos, ByteBuffer buffer, RunnableWithParameter<Pair<Integer,IOException>> ondone) Same as IO.Readable.readAsync(ByteBuffer, RunnableWithParameter) but read at the given position.
int	readFully(byte[] buffer) Read bytes to fill the given buffer, and return the number of bytes read which is less than the size of the buffer only if the end of the IO is reached, and so may return 0.
AsyncWork<Integer,IOException>	readFullyAsync(ByteBuffer buffer, RunnableWithParameter<Pair<Integer,IOException>> ondone) Asynchronous read operation to fully fill the given buffer.
AsyncWork<Integer,IOException>	readFullyAsync(long pos, ByteBuffer buffer, RunnableWithParameter<Pair<Integer,IOException>> ondone) Same as IO.Readable.readFullyAsync(ByteBuffer, RunnableWithParameter) but read at the given position.
int	readFullySync(ByteBuffer buffer) Synchronous read operation to fully fill the given buffer.
int	readFullySync(long pos, ByteBuffer buffer) Same as IO.Readable.readFullySync(ByteBuffer) but read at the given position.
AsyncWork<Integer,IOException>	readFullySyncIfPossible(ByteBuffer buffer, RunnableWithParameter<Pair<Integer,IOException>> ondone) While readAsync methods are supposed to do the job in a separate thread, this method fills the given buffer synchronously if enough data is already buffered, else it finishes asynchronously.
AsyncWork<Integer,IOException>	readFullySyncIfPossible(long pos, ByteBuffer buffer, RunnableWithParameter<Pair<Integer,IOException>> ondone) While readAsync methods are supposed to do the job in a separate thread, this method fills the given buffer synchronously if enough data is already buffered, else it finishes asynchronously.
AsyncWork<ByteBuffer,IOException>	readNextBufferAsync(RunnableWithParameter<Pair<ByteBuffer,IOException>> ondone) Retrieve a buffer of bytes in the most efficient way depending on the implementation of Buffered. This method is similar to readAsync, it read some bytes that are immediately available with a minimum of operations.
int	readSync(ByteBuffer buffer) Read synchronously into the given buffer.
int	readSync(long pos, ByteBuffer buffer) Same as IO.Readable.readSync(ByteBuffer) but read at the given position.
AsyncWork<Long,IOException>	seekAsync(IO.Seekable.SeekType type, long move, RunnableWithParameter<Pair<Long,IOException>> ondone) Returns the new position.
long	seekSync(IO.Seekable.SeekType type, long move) Returns the new position.
void	setPriority(byte priority) Set the priority of asynchronous operations.
int	skip(int skip) Same as IO.Readable.skipSync(long) but limited to an integer.
AsyncWork<Long,IOException>	skipAsync(long n, RunnableWithParameter<Pair<Long,IOException>> ondone) Returns the number of bytes skipped.
long	skipSync(long n) Returns the number of bytes skipped.

Methods inherited from class net.lecousin.framework.util.ConcurrentCloseable

addCloseListener, addCloseListener, close, closeAsync, isClosed, isClosing, lockClose, operation, operation, removeCloseListener, removeCloseListener, unlockClose

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface net.lecousin.framework.io.IO.Readable.Seekable

readAsync, readFullyAsync

Methods inherited from interface net.lecousin.framework.io.IO.Seekable

seekAsync

Methods inherited from interface net.lecousin.framework.io.IO

error

Methods inherited from interface net.lecousin.framework.util.IConcurrentCloseable

lockClose, unlockClose

Methods inherited from interface java.lang.AutoCloseable

close

Methods inherited from interface net.lecousin.framework.util.AsyncCloseable

closeAsync

Methods inherited from interface net.lecousin.framework.util.CloseableListenable

addCloseListener, addCloseListener, close, isClosed, removeCloseListener, removeCloseListener

Methods inherited from interface net.lecousin.framework.io.IO.Readable.Buffered

readFullySyncIfPossible, readNextBufferAsync

Methods inherited from interface net.lecousin.framework.io.IO.Readable

readAsync, readFullyAsync, skipAsync

Methods inherited from interface net.lecousin.framework.io.IO.ReadableByteStream

readByte

Field Detail

memory

protected BufferedIO.MemoryManagement memory

io

protected IO.Readable.Seekable io

position

protected long position

size

protected long size

closing

protected boolean closing

bufferTable

protected net.lecousin.framework.io.buffering.BufferedIO.BufferTable bufferTable

preLoadNextBuffer

protected boolean preLoadNextBuffer

firstBufferSize

protected int firstBufferSize

bufferSize

protected int bufferSize

Constructor Detail

BufferedIO

public BufferedIO(IO.Readable.Seekable io,
                  long size,
                  int firstBufferSize,
                  int nextBuffersSize,
                  boolean preLoadNextBuffer)

Constructor.

BufferedIO

public BufferedIO(IO.Readable.Seekable io,
                  int firstBufferSize,
                  int nextBuffersSize,
                  boolean preLoadNextBuffer)
           throws IOException

Constructor.

Throws:

IOException

Method Detail

getBufferIndex

protected long getBufferIndex(long pos)

getBufferOffset

protected int getBufferOffset(long pos)

getBufferPosition

protected long getBufferPosition(long index)

preLoadBuffer

protected void preLoadBuffer(long pos)

closeUnderlyingResources

protected ISynchronizationPoint<Exception> closeUnderlyingResources()

Specified by:

closeUnderlyingResources in class ConcurrentCloseable

closeResources

protected void closeResources(SynchronizationPoint<Exception> ondone)

Specified by:

closeResources in class ConcurrentCloseable

getPriority

public byte getPriority()

Description copied from class: ConcurrentCloseable

Return the priority.

Specified by:

getPriority in interface IO

Specified by:

getPriority in class ConcurrentCloseable

setPriority

public void setPriority(byte priority)

Description copied from interface: IO

Set the priority of asynchronous operations.

Specified by:

setPriority in interface IO

getSourceDescription

public String getSourceDescription()

Description copied from interface: IO

Describe the IO, which can be used for logging or debugging purposes.

Specified by:

getSourceDescription in interface IO

getWrappedIO

public IO getWrappedIO()

Description copied from interface: IO

If this IO is wrapping another one (such as adding Buffered capabilities), this method returns it, or null if it does not wrap another IO.

Specified by:

getWrappedIO in interface IO

getTaskManager

public TaskManager getTaskManager()

Description copied from interface: IO

Return the TaskManager used for asynchronous operations.

Specified by:

getTaskManager in interface IO

getSizeSync

public long getSizeSync()

Description copied from interface: IO.KnownSize

Synchronous operation to get the size of the IO.

Specified by:

getSizeSync in interface IO.KnownSize

getSizeAsync

public AsyncWork<Long,IOException> getSizeAsync()

Description copied from interface: IO.KnownSize

Asynchronous operation to get the size of the IO.

Specified by:

getSizeAsync in interface IO.KnownSize

getPosition

public long getPosition()

Description copied from interface: IO.PositionKnown

Return the current position.

Specified by:

getPosition in interface IO.PositionKnown

read

public int read()
         throws IOException

Description copied from interface: IO.ReadableByteStream

Read next byte. Return -1 if the end of the IO is reached, or the value between 0 and 255.

Specified by:

read in interface IO.ReadableByteStream

Throws:

IOException

read

public int read(byte[] buffer,
                int offset,
                int len)
         throws IOException

Description copied from interface: IO.ReadableByteStream

Read bytes into the given buffer, starting to store them at the given offset, and with the given maximum number of bytes. Returns the number of bytes read, or 0 or -1 if the end of the IO is reached.

Specified by:

read in interface IO.ReadableByteStream

Throws:

IOException

readSync

public int readSync(long pos,
                    ByteBuffer buffer)
             throws IOException

Description copied from interface: IO.Readable.Seekable

Same as IO.Readable.readSync(ByteBuffer) but read at the given position. The current position is changed to the given position plus the number of bytes read.

Specified by:

readSync in interface IO.Readable.Seekable

Throws:

IOException

readSync

public int readSync(ByteBuffer buffer)
             throws IOException

Description copied from interface: IO.Readable

Read synchronously into the given buffer. The buffer may not be filled, but at least one byte is read. If there is no available byte, this method is blocking.

Specified by:

readSync in interface IO.Readable

Parameters:

buffer - buffer to fill

Returns:

the number of bytes read, or 0 or -1 if the end of the IO is reached.

Throws:

IOException

readFullySync

public int readFullySync(long pos,
                         ByteBuffer buffer)
                  throws IOException

Description copied from interface: IO.Readable.Seekable

Same as IO.Readable.readFullySync(ByteBuffer) but read at the given position. The current position is changed to the given position plus the number of bytes read.

Specified by:

readFullySync in interface IO.Readable.Seekable

Throws:

IOException

readFullySync

public int readFullySync(ByteBuffer buffer)
                  throws IOException

Description copied from interface: IO.Readable

Synchronous read operation to fully fill the given buffer. This method does not return until the buffer has been fully filled, or the end of the IO has been reached.

Specified by:

readFullySync in interface IO.Readable

Parameters:

buffer - the buffer to fill

Returns:

the number of bytes read, possibly 0 if the end of the IO has been reached.

Throws:

IOException

readFully

public int readFully(byte[] buffer)
              throws IOException

Description copied from interface: IO.ReadableByteStream

Read bytes to fill the given buffer, and return the number of bytes read which is less than the size of the buffer only if the end of the IO is reached, and so may return 0.

Specified by:

readFully in interface IO.ReadableByteStream

Throws:

IOException

canStartReading

public ISynchronizationPoint<IOException> canStartReading()

Description copied from interface: IO.Readable

Return a synchronization point that is unblocked when data is ready to be read. This allows to start reading operations only when we know it won't block.

Specified by:

canStartReading in interface IO.Readable

Specified by:

canStartReading in interface IO.ReadableByteStream

readAsync

public int readAsync()
              throws IOException

Description copied from interface: IO.Readable.Buffered

Read a single byte if possible.

Specified by:

readAsync in interface IO.Readable.Buffered

Returns:

the next byte, or -1 if the end of the IO has been reached, or -2 if no more byte is available.

Throws:

IOException

readAsync

public AsyncWork<Integer,IOException> readAsync(long pos,
                                                ByteBuffer buffer,
                                                RunnableWithParameter<Pair<Integer,IOException>> ondone)

Description copied from interface: IO.Readable.Seekable

Same as IO.Readable.readAsync(ByteBuffer, RunnableWithParameter) but read at the given position. The current position is changed to the given position plus the number of bytes read upon completion.

Specified by:

readAsync in interface IO.Readable.Seekable

readAsync

public AsyncWork<Integer,IOException> readAsync(ByteBuffer buffer,
                                                RunnableWithParameter<Pair<Integer,IOException>> ondone)

Description copied from interface: IO.Readable

Asynchronous read operation. The buffer may not be filled, but at least one byte is read. If there is no available byte, the returned AsyncWork won't be unblocked until at least one byte is available, or we know that the end of the IO has been reached.

Specified by:

readAsync in interface IO.Readable

Parameters:

buffer - buffer to fill

ondone - called before the returned AsyncWork is unblocked and its listeners are called.

Returns:

the Integer contains the number of bytes read, or 0 or -1 if the end of the IO is reached.

readNextBufferAsync

public AsyncWork<ByteBuffer,IOException> readNextBufferAsync(RunnableWithParameter<Pair<ByteBuffer,IOException>> ondone)

Description copied from interface: IO.Readable.Buffered

Retrieve a buffer of bytes in the most efficient way depending on the implementation of Buffered.
This method is similar to readAsync, it read some bytes that are immediately available with a minimum of operations. The problem with readAsync is that we have to give a buffer to fill. If there are more bytes that can be read, a new call will be needed. If there not enough available bytes, the given buffer is not fully filled and uses more memory than needed.
This method is more efficient because it leaves the implementation allocating a buffer with a size that can be fully filled in a minimum number of operations.
In addition, using readAsync needs to keep the given buffer in a variable or in the closure, while using this method the buffer is directly given as the result of the operation.
The returned buffer is ready to be read (no need to flip), and the number of bytes read can be obtain using the method remaining of the buffer.

Specified by:

readNextBufferAsync in interface IO.Readable.Buffered

readFullySyncIfPossible

public AsyncWork<Integer,IOException> readFullySyncIfPossible(ByteBuffer buffer,
                                                              RunnableWithParameter<Pair<Integer,IOException>> ondone)

Description copied from interface: IO.Readable.Buffered

While readAsync methods are supposed to do the job in a separate thread, this method fills the given buffer synchronously if enough data is already buffered, else it finishes asynchronously. The caller can check the returned AsyncWork by calling its method isUnblocked to know if the read has been performed synchronously. This method may be useful for processes that hope to work synchronously because this IO is buffered, but support also to work asynchronously without blocking a thread.

Specified by:

readFullySyncIfPossible in interface IO.Readable.Buffered

readFullySyncIfPossible

public AsyncWork<Integer,IOException> readFullySyncIfPossible(long pos,
                                                              ByteBuffer buffer,
                                                              RunnableWithParameter<Pair<Integer,IOException>> ondone)

While readAsync methods are supposed to do the job in a separate thread, this method fills the given buffer synchronously if enough data is already buffered, else it finishes asynchronously. The caller can check the returned AsyncWork by calling its method isUnblocked to know if the read has been performed synchronously. This method may be useful for processes that hope to work synchronously because this IO is buffered, but support also to work asynchronously without blocking a thread.

readFullyAsync

public AsyncWork<Integer,IOException> readFullyAsync(long pos,
                                                     ByteBuffer buffer,
                                                     RunnableWithParameter<Pair<Integer,IOException>> ondone)

Description copied from interface: IO.Readable.Seekable

Same as IO.Readable.readFullyAsync(ByteBuffer, RunnableWithParameter) but read at the given position. The current position is changed to the given position plus the number of bytes read upon completion.

Specified by:

readFullyAsync in interface IO.Readable.Seekable

readFullyAsync

public AsyncWork<Integer,IOException> readFullyAsync(ByteBuffer buffer,
                                                     RunnableWithParameter<Pair<Integer,IOException>> ondone)

Description copied from interface: IO.Readable

Asynchronous read operation to fully fill the given buffer. The returned AsyncWork is not unblocked until the buffer has been fully filled, or the end of the IO has been reached.

Specified by:

readFullyAsync in interface IO.Readable

Parameters:

buffer - the buffer to fill

ondone - called before the returned AsyncWork is unblocked and its listeners are called.

Returns:

the Integer contains the number of bytes read, possibly 0 if the end of the IO has been reached.

skipSync

public long skipSync(long n)

Description copied from interface: IO.Readable

Returns the number of bytes skipped.

Specified by:

skipSync in interface IO.Readable

skip

public int skip(int skip)

Description copied from interface: IO.ReadableByteStream

Same as IO.Readable.skipSync(long) but limited to an integer.

Specified by:

skip in interface IO.ReadableByteStream

skipAsync

public AsyncWork<Long,IOException> skipAsync(long n,
                                             RunnableWithParameter<Pair<Long,IOException>> ondone)

Description copied from interface: IO.Readable

Returns the number of bytes skipped.

Specified by:

skipAsync in interface IO.Readable

seekSync

public long seekSync(IO.Seekable.SeekType type,
                     long move)

Description copied from interface: IO.Seekable

Returns the new position. The implementation must not allow to seek after the end of the file, the method setSize should be used instead. It must accept seeking beyond the end, but position at the end, and accept seeking before the beginning, but position at the beginning.

Specified by:

seekSync in interface IO.Seekable

seekAsync

public AsyncWork<Long,IOException> seekAsync(IO.Seekable.SeekType type,
                                             long move,
                                             RunnableWithParameter<Pair<Long,IOException>> ondone)

Description copied from interface: IO.Seekable

Returns the new position. The implementation must not allow to seek after the end of the file, the method setSize should be used instead. It must accept seeking beyond the end, but position at the end, and accept seeking before the beginning, but position at the beginning.

Specified by:

seekAsync in interface IO.Seekable