org.apache.hadoop.fs.s3a

Class S3AInputStream

java.lang.Object

java.io.InputStream

org.apache.hadoop.fs.FSInputStream

org.apache.hadoop.fs.s3a.S3AInputStream

All Implemented Interfaces:

Closeable, AutoCloseable, org.apache.hadoop.fs.CanSetReadahead, org.apache.hadoop.fs.PositionedReadable, org.apache.hadoop.fs.Seekable

@InterfaceAudience.Private
@InterfaceStability.Evolving
public class S3AInputStream
extends org.apache.hadoop.fs.FSInputStream
implements org.apache.hadoop.fs.CanSetReadahead

The input stream for an S3A object. As this stream seeks withing an object, it may close then re-open the stream. When this happens, any updated stream data may be retrieved, and, given the consistency model of Amazon S3, outdated data may in fact be picked up. As a result, the outcome of reading from a stream of an object which is actively manipulated during the read process is "undefined". The class is marked as private as code should not be creating instances themselves. Any extra feature (e.g instrumentation) should be considered unstable. Because it prints some of the state of the instrumentation, the output of toString() must also be considered unstable.

Constructor Summary

Constructors

Constructor and Description
S3AInputStream(org.apache.hadoop.fs.s3a.S3ObjectAttributes s3Attributes, long contentLength, com.amazonaws.services.s3.AmazonS3 client, org.apache.hadoop.fs.FileSystem.Statistics stats, S3AInstrumentation instrumentation, long readahead, S3AInputPolicy inputPolicy)

Method Summary

Methods

Modifier and Type	Method and Description
int	available()
void	close() Close the stream.
long	getContentRangeFinish()
long	getContentRangeStart()
long	getPos()
long	getReadahead() Get the current readahead value.
S3AInstrumentation.InputStreamStatistics	getS3AStreamStatistics() Access the input stream statistics.
boolean	markSupported()
int	read()
int	read(byte[] buf, int off, int len) This updates the statistics on read operations started and whether or not the read operation "completed", that is: returned the exact number of bytes requested.
void	readFully(long position, byte[] buffer, int offset, int length) Subclass readFully() operation which only seeks at the start of the series of operations; seeking back at the end.
long	remainingInCurrentRequest() Bytes left in the current request.
long	remainingInFile() Bytes left in stream.
boolean	resetConnection() Forcibly reset the stream, by aborting the connection.
void	seek(long targetPos)
boolean	seekToNewSource(long targetPos)
void	setReadahead(Long readahead)
String	toString() String value includes statistics as well as stream state.

Methods inherited from class org.apache.hadoop.fs.FSInputStream

read, readFully, validatePositionedReadArgs

Methods inherited from class java.io.InputStream

mark, read, reset, skip

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

S3AInputStream

public S3AInputStream(org.apache.hadoop.fs.s3a.S3ObjectAttributes s3Attributes,
              long contentLength,
              com.amazonaws.services.s3.AmazonS3 client,
              org.apache.hadoop.fs.FileSystem.Statistics stats,
              S3AInstrumentation instrumentation,
              long readahead,
              S3AInputPolicy inputPolicy)

Method Detail

getPos

public long getPos()
            throws IOException

Specified by:

getPos in interface org.apache.hadoop.fs.Seekable

Specified by:

getPos in class org.apache.hadoop.fs.FSInputStream

Throws:

IOException

seek

public void seek(long targetPos)
          throws IOException

Specified by:

seek in interface org.apache.hadoop.fs.Seekable

Specified by:

seek in class org.apache.hadoop.fs.FSInputStream

Throws:

IOException

seekToNewSource

public boolean seekToNewSource(long targetPos)
                        throws IOException

Specified by:

seekToNewSource in interface org.apache.hadoop.fs.Seekable

Specified by:

seekToNewSource in class org.apache.hadoop.fs.FSInputStream

Throws:

IOException

read

public int read()
         throws IOException

Specified by:

read in class InputStream

Throws:

IOException

read

public int read(byte[] buf,
       int off,
       int len)
         throws IOException

This updates the statistics on read operations started and whether or not the read operation "completed", that is: returned the exact number of bytes requested.

Overrides:

read in class InputStream

Throws:

IOException - if there are other problems

close

public void close()
           throws IOException

Close the stream. This triggers publishing of the stream statistics back to the filesystem statistics. This operation is synchronized, so that only one thread can attempt to close the connection; all later/blocked calls are no-ops.

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Overrides:

close in class InputStream

Throws:

IOException - on any problem

resetConnection

@InterfaceStability.Unstable
public boolean resetConnection()
                        throws IOException

Forcibly reset the stream, by aborting the connection. The next read() operation will trigger the opening of a new HTTPS connection. This is potentially very inefficient, and should only be invoked in extreme circumstances. It logs at info for this reason.

Returns:

true if the connection was actually reset.

Throws:

IOException - if invoked on a closed stream.

available

public int available()
              throws IOException

Overrides:

available in class InputStream

Throws:

IOException

remainingInFile

@InterfaceAudience.Private
@InterfaceStability.Unstable
public long remainingInFile()

Bytes left in stream.

Returns:

how many bytes are left to read

remainingInCurrentRequest

@InterfaceAudience.Private
@InterfaceStability.Unstable
public long remainingInCurrentRequest()

Bytes left in the current request. Only valid if there is an active request.

Returns:

how many bytes are left to read in the current GET.

getContentRangeFinish

@InterfaceAudience.Private
@InterfaceStability.Unstable
public long getContentRangeFinish()

getContentRangeStart

@InterfaceAudience.Private
@InterfaceStability.Unstable
public long getContentRangeStart()

markSupported

public boolean markSupported()

Overrides:

markSupported in class InputStream

toString

@InterfaceStability.Unstable
public String toString()

String value includes statistics as well as stream state. Important: there are no guarantees as to the stability of this value.

Overrides:

toString in class Object

Returns:

a string value for printing in logs/diagnostics

readFully

public void readFully(long position,
             byte[] buffer,
             int offset,
             int length)
               throws IOException

Subclass readFully() operation which only seeks at the start of the series of operations; seeking back at the end. This is significantly higher performance if multiple read attempts are needed to fetch the data, as it does not break the HTTP connection. To maintain thread safety requirements, this operation is synchronized for the duration of the sequence.

Specified by:

readFully in interface org.apache.hadoop.fs.PositionedReadable

Overrides:

readFully in class org.apache.hadoop.fs.FSInputStream

Throws:

IOException

getS3AStreamStatistics

@InterfaceAudience.Private
@InterfaceStability.Unstable
public S3AInstrumentation.InputStreamStatistics getS3AStreamStatistics()

Access the input stream statistics. This is for internal testing and may be removed without warning.

Returns:

the statistics for this input stream

setReadahead

public void setReadahead(Long readahead)

Specified by:

setReadahead in interface org.apache.hadoop.fs.CanSetReadahead

getReadahead

public long getReadahead()

Get the current readahead value.

Returns:

a non-negative readahead value