org.apache.hadoop.hbase.io.asyncfs

Class FanOutOneBlockAsyncDFSOutput

java.lang.Object

org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput

All Implemented Interfaces:

Closeable, AutoCloseable, AsyncFSOutput

@InterfaceAudience.Private
public class FanOutOneBlockAsyncDFSOutput
extends Object
implements AsyncFSOutput

An asynchronous HDFS output stream implementation which fans out data to datanode and only supports writing file with only one block.

Use the createOutput method in FanOutOneBlockAsyncDFSOutputHelper to create. The mainly usage of this class is implementing WAL, so we only expose a little HDFS configurations in the method. And we place it here under util package because we want to make it independent of WAL implementation thus easier to move it to HDFS project finally.

Note that, all connections to datanode will run in the same EventLoop which means we only need one thread here. But be careful, we do some blocking operations in close() and recoverAndClose(CancelableProgressable) methods, so do not call them inside EventLoop. And for write(byte[]) write(byte[], int, int), buffered() and flush(boolean), if you call them outside EventLoop, there will be an extra context-switch.

Advantages compare to DFSOutputStream:

1. The fan out mechanism. This will reduce the latency.
2. The asynchronous WAL could also run in the same EventLoop, we could just call write and flush inside the EventLoop thread, so generally we only have one thread to do all the things.
3. Fail-fast when connection to datanode error. The WAL implementation could open new writer ASAP.
4. We could benefit from netty's ByteBuf management mechanism.

Method Summary

Modifier and Type	Method and Description
int	buffered() Return the current size of buffered data.
void	close() End the current block and complete file at namenode.
CompletableFuture<Long>	flush(boolean syncBlock) Flush the buffer out to datanodes.
org.apache.hadoop.hdfs.protocol.DatanodeInfo[]	getPipeline() Return current pipeline.
void	recoverAndClose(CancelableProgressable reporter) The close method when error occurred.
void	write(byte[] b) Just call write(b, 0, b.length).
void	write(byte[] b, int off, int len) Copy the data into the buffer.
void	write(ByteBuffer bb) Copy the data in the given bb into the buffer.
void	writeInt(int i) Write an int to the buffer.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

writeInt

public void writeInt(int i)

Description copied from interface: AsyncFSOutput

Write an int to the buffer.

Specified by:

writeInt in interface AsyncFSOutput

write

public void write(ByteBuffer bb)

Description copied from interface: AsyncFSOutput

Copy the data in the given bb into the buffer.

Specified by:

write in interface AsyncFSOutput

write

public void write(byte[] b)

Description copied from interface: AsyncFSOutput

Just call write(b, 0, b.length).

Specified by:

write in interface AsyncFSOutput

See Also:

AsyncFSOutput.write(byte[], int, int)

write

public void write(byte[] b,
                  int off,
                  int len)

Description copied from interface: AsyncFSOutput

Copy the data into the buffer. Note that you need to call AsyncFSOutput.flush(boolean) to flush the buffer manually.

Specified by:

write in interface AsyncFSOutput

buffered

public int buffered()

Description copied from interface: AsyncFSOutput

Return the current size of buffered data.

Specified by:

buffered in interface AsyncFSOutput

getPipeline

public org.apache.hadoop.hdfs.protocol.DatanodeInfo[] getPipeline()

Description copied from interface: AsyncFSOutput

Return current pipeline. Empty array if no pipeline.

Specified by:

getPipeline in interface AsyncFSOutput

flush

public CompletableFuture<Long> flush(boolean syncBlock)

Flush the buffer out to datanodes.

Specified by:

flush in interface AsyncFSOutput

Parameters:

syncBlock - will call hsync if true, otherwise hflush.

Returns:

A CompletableFuture that hold the acked length after flushing.

recoverAndClose

public void recoverAndClose(CancelableProgressable reporter)
                     throws IOException

The close method when error occurred. Now we just call recoverFileLease.

Specified by:

recoverAndClose in interface AsyncFSOutput

Throws:

IOException

close

public void close()
           throws IOException

End the current block and complete file at namenode. You should call recoverAndClose(CancelableProgressable) if this method throws an exception.

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Specified by:

close in interface AsyncFSOutput

Throws:

IOException