com.google.cloud.hadoop.fs.gcs

Class GoogleHadoopSyncableOutputStream

java.lang.Object

java.io.OutputStream

com.google.cloud.hadoop.fs.gcs.GoogleHadoopSyncableOutputStream

All Implemented Interfaces:

Closeable, Flushable, AutoCloseable, org.apache.hadoop.fs.Syncable

public class GoogleHadoopSyncableOutputStream
extends OutputStream
implements org.apache.hadoop.fs.Syncable

GoogleHadoopSyncableOutputStream implements the Syncable interface by composing objects created in separate underlying streams for each hsync() call.

Prior to the first hsync(), sync() or close() call, this channel will behave the same way as a basic non-syncable channel, writing directly to the destination file.

On the first call to hsync()/sync(), the destination file is committed and a new temporary file using a hidden-file prefix (underscore) is created with an additional suffix which differs for each subsequent temporary file in the series; during this time readers can read the data committed to the destination file, but not the bytes written to the temporary file since the last hsync() call.

On each subsequent hsync()/sync() call, the temporary file closed(), composed onto the destination file, then deleted, and a new temporary file is opened under a new filename for further writes.

Caveat: each hsync()/sync() requires many underlying read and mutation requests occurring sequentially, so latency is expected to be fairly high.

If errors occur mid-stream, there may be one or more temporary files failing to be cleaned up, and require manual intervention to discover and delete any such unused files. Data written prior to the most recent successful hsync() is persistent and safe in such a case.

If multiple writers are attempting to write to the same destination file, generation ids used with low-level precondition checks will cause all but a one writer to fail their precondition checks during writes, and a single remaining writer will safely occupy the stream.

Field Summary

Fields

Modifier and Type	Field and Description
static String	TEMPFILE_PREFIX

Constructor Summary

Constructors

Constructor and Description
GoogleHadoopSyncableOutputStream(GoogleHadoopFileSystemBase ghfs, URI gcsPath, org.apache.hadoop.fs.FileSystem.Statistics statistics, CreateFileOptions createFileOptions) Deprecated.
GoogleHadoopSyncableOutputStream(GoogleHadoopFileSystemBase ghfs, URI gcsPath, org.apache.hadoop.fs.FileSystem.Statistics statistics, CreateFileOptions createFileOptions, boolean appendMode) Deprecated.
GoogleHadoopSyncableOutputStream(GoogleHadoopFileSystemBase ghfs, URI gcsPath, org.apache.hadoop.fs.FileSystem.Statistics statistics, CreateFileOptions createFileOptions, SyncableOutputStreamOptions options) Creates a new GoogleHadoopSyncableOutputStream.

Method Summary

Modifier and Type	Method and Description
void	close()
void	hflush() There is no way to flush data to become available for readers without a full-fledged hsync(), If the output stream is only syncable, this method is a no-op.
void	hsync()
void	sync()
void	write(byte[] b, int offset, int len)
void	write(int b)

Methods inherited from class java.io.OutputStream

flush, write

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

TEMPFILE_PREFIX

public static final String TEMPFILE_PREFIX

See Also:

Constant Field Values

Constructor Detail

GoogleHadoopSyncableOutputStream

@Deprecated
public GoogleHadoopSyncableOutputStream(GoogleHadoopFileSystemBase ghfs,
                                                    URI gcsPath,
                                                    org.apache.hadoop.fs.FileSystem.Statistics statistics,
                                                    CreateFileOptions createFileOptions)
                                             throws IOException

Deprecated.

Creates a new GoogleHadoopSyncableOutputStream with initial stream initialized and expected to begin at file-offset 0. This constructor is not suitable for "appending" to already existing files.

Throws:

IOException

GoogleHadoopSyncableOutputStream

@Deprecated
public GoogleHadoopSyncableOutputStream(GoogleHadoopFileSystemBase ghfs,
                                                    URI gcsPath,
                                                    org.apache.hadoop.fs.FileSystem.Statistics statistics,
                                                    CreateFileOptions createFileOptions,
                                                    boolean appendMode)
                                             throws IOException

Deprecated.

Creates a new GoogleHadoopSyncableOutputStream suitable for appending to existing files.

Throws:

IOException

GoogleHadoopSyncableOutputStream

public GoogleHadoopSyncableOutputStream(GoogleHadoopFileSystemBase ghfs,
                                        URI gcsPath,
                                        org.apache.hadoop.fs.FileSystem.Statistics statistics,
                                        CreateFileOptions createFileOptions,
                                        SyncableOutputStreamOptions options)
                                 throws IOException

Creates a new GoogleHadoopSyncableOutputStream.

Throws:

IOException

Method Detail

write

public void write(int b)
           throws IOException

Specified by:

write in class OutputStream

Throws:

IOException

write

public void write(byte[] b,
                  int offset,
                  int len)
           throws IOException

Overrides:

write in class OutputStream

Throws:

IOException

close

public void close()
           throws IOException

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Overrides:

close in class OutputStream

Throws:

IOException

sync

public void sync()
          throws IOException

Specified by:

sync in interface org.apache.hadoop.fs.Syncable

Throws:

IOException

hflush

public void hflush()
            throws IOException

There is no way to flush data to become available for readers without a full-fledged hsync(), If the output stream is only syncable, this method is a no-op. If the output stream is also flushable, this method will simply use the same implementation of hsync().

If it is rate limited, unlike hsync(), which will try to acquire the permits and block, it will do nothing.

Specified by:

hflush in interface org.apache.hadoop.fs.Syncable

Throws:

IOException

hsync

public void hsync()
           throws IOException

Specified by:

hsync in interface org.apache.hadoop.fs.Syncable

Throws:

IOException