org.apache.hadoop.fs.s3a

Class S3AInputStream

java.lang.Object

java.io.InputStream

org.apache.hadoop.fs.FSInputStream

org.apache.hadoop.fs.s3a.S3AInputStream

All Implemented Interfaces:

Closeable, AutoCloseable, org.apache.hadoop.fs.CanSetReadahead, org.apache.hadoop.fs.CanUnbuffer, org.apache.hadoop.fs.PositionedReadable, org.apache.hadoop.fs.Seekable, org.apache.hadoop.fs.StreamCapabilities

@InterfaceAudience.Private
 @InterfaceStability.Evolving
public class S3AInputStream
extends org.apache.hadoop.fs.FSInputStream
implements org.apache.hadoop.fs.CanSetReadahead, org.apache.hadoop.fs.CanUnbuffer, org.apache.hadoop.fs.StreamCapabilities

The input stream for an S3A object. As this stream seeks withing an object, it may close then re-open the stream. When this happens, any updated stream data may be retrieved, and, given the consistency model of Amazon S3, outdated data may in fact be picked up. As a result, the outcome of reading from a stream of an object which is actively manipulated during the read process is "undefined". The class is marked as private as code should not be creating instances themselves. Any extra feature (e.g instrumentation) should be considered unstable. Because it prints some of the state of the instrumentation, the output of toString() must also be considered unstable.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.hadoop.fs.StreamCapabilities

org.apache.hadoop.fs.StreamCapabilities.StreamCapability

Field Summary

Fields

Modifier and Type	Field and Description
static String	E_NEGATIVE_READAHEAD_VALUE
static String	OPERATION_OPEN
static String	OPERATION_REOPEN

Fields inherited from interface org.apache.hadoop.fs.StreamCapabilities

DROPBEHIND, HFLUSH, HSYNC, PREADBYTEBUFFER, READAHEAD, READBYTEBUFFER, UNBUFFER

Constructor Summary

Constructors

Constructor and Description
S3AInputStream(S3AReadOpContext ctx, S3ObjectAttributes s3Attributes, com.amazonaws.services.s3.AmazonS3 client) Create the stream.

Method Summary

Modifier and Type	Method and Description
int	available()
void	close() Close the stream.
long	getContentRangeFinish()
long	getContentRangeStart()
long	getPos()
long	getReadahead() Get the current readahead value.
S3AInstrumentation.InputStreamStatistics	getS3AStreamStatistics() Access the input stream statistics.
boolean	hasCapability(String capability)
boolean	markSupported()
int	read()
int	read(byte[] buf, int off, int len) This updates the statistics on read operations started and whether or not the read operation "completed", that is: returned the exact number of bytes requested.
void	readFully(long position, byte[] buffer, int offset, int length) Subclass readFully() operation which only seeks at the start of the series of operations; seeking back at the end.
long	remainingInCurrentRequest() Bytes left in the current request.
long	remainingInFile() Bytes left in stream.
boolean	resetConnection() Forcibly reset the stream, by aborting the connection.
void	seek(long targetPos)
boolean	seekToNewSource(long targetPos)
void	setReadahead(Long readahead)
String	toString() String value includes statistics as well as stream state.
void	unbuffer() Closes the underlying S3 stream, and merges the streamStatistics instance associated with the stream.
static long	validateReadahead(Long readahead) from a possibly null Long value, return a valid readahead.

Methods inherited from class org.apache.hadoop.fs.FSInputStream

read, readFully, validatePositionedReadArgs

Methods inherited from class java.io.InputStream

mark, read, reset, skip

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

E_NEGATIVE_READAHEAD_VALUE

public static final String E_NEGATIVE_READAHEAD_VALUE

See Also:

Constant Field Values

OPERATION_OPEN

public static final String OPERATION_OPEN

See Also:

Constant Field Values

OPERATION_REOPEN

public static final String OPERATION_REOPEN

See Also:

Constant Field Values

Constructor Detail

S3AInputStream

public S3AInputStream(S3AReadOpContext ctx,
                      S3ObjectAttributes s3Attributes,
                      com.amazonaws.services.s3.AmazonS3 client)

Create the stream. This does not attempt to open it; that is only done on the first actual read() operation.

Parameters:

ctx - operation context

s3Attributes - object attributes

client - S3 client to use

Method Detail

getPos

public long getPos()
            throws IOException

Specified by:

getPos in interface org.apache.hadoop.fs.Seekable

Specified by:

getPos in class org.apache.hadoop.fs.FSInputStream

Throws:

IOException

seek

public void seek(long targetPos)
          throws IOException

Specified by:

seek in interface org.apache.hadoop.fs.Seekable

Specified by:

seek in class org.apache.hadoop.fs.FSInputStream

Throws:

IOException

seekToNewSource

public boolean seekToNewSource(long targetPos)
                        throws IOException

Specified by:

seekToNewSource in interface org.apache.hadoop.fs.Seekable

Specified by:

seekToNewSource in class org.apache.hadoop.fs.FSInputStream

Throws:

IOException

read

@Retries.RetryTranslated
public int read()
                                  throws IOException

Specified by:

read in class InputStream

Throws:

IOException

read

@Retries.RetryTranslated
public int read(byte[] buf,
                                         int off,
                                         int len)
                                  throws IOException

This updates the statistics on read operations started and whether or not the read operation "completed", that is: returned the exact number of bytes requested.

Overrides:

read in class InputStream

Throws:

IOException - if there are other problems

close

public void close()
           throws IOException

Close the stream. This triggers publishing of the stream statistics back to the filesystem statistics. This operation is synchronized, so that only one thread can attempt to close the connection; all later/blocked calls are no-ops.

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Overrides:

close in class InputStream

Throws:

IOException - on any problem

resetConnection

@InterfaceStability.Unstable
public boolean resetConnection()
                                                     throws IOException

Forcibly reset the stream, by aborting the connection. The next read() operation will trigger the opening of a new HTTPS connection. This is potentially very inefficient, and should only be invoked in extreme circumstances. It logs at info for this reason.

Returns:

true if the connection was actually reset.

Throws:

IOException - if invoked on a closed stream.

available

public int available()
              throws IOException

Overrides:

available in class InputStream

Throws:

IOException

remainingInFile

@InterfaceAudience.Private
 @InterfaceStability.Unstable
public long remainingInFile()

Bytes left in stream.

Returns:

how many bytes are left to read

remainingInCurrentRequest

@InterfaceAudience.Private
 @InterfaceStability.Unstable
public long remainingInCurrentRequest()

Bytes left in the current request. Only valid if there is an active request.

Returns:

how many bytes are left to read in the current GET.

getContentRangeFinish

@InterfaceAudience.Private
 @InterfaceStability.Unstable
public long getContentRangeFinish()

getContentRangeStart

@InterfaceAudience.Private
 @InterfaceStability.Unstable
public long getContentRangeStart()

markSupported

public boolean markSupported()

Overrides:

markSupported in class InputStream

toString

@InterfaceStability.Unstable
public String toString()

String value includes statistics as well as stream state. Important: there are no guarantees as to the stability of this value.

Overrides:

toString in class Object

Returns:

a string value for printing in logs/diagnostics

readFully

@Retries.RetryTranslated
public void readFully(long position,
                                               byte[] buffer,
                                               int offset,
                                               int length)
                                        throws IOException

Subclass readFully() operation which only seeks at the start of the series of operations; seeking back at the end. This is significantly higher performance if multiple read attempts are needed to fetch the data, as it does not break the HTTP connection. To maintain thread safety requirements, this operation is synchronized for the duration of the sequence.

Specified by:

readFully in interface org.apache.hadoop.fs.PositionedReadable

Overrides:

readFully in class org.apache.hadoop.fs.FSInputStream

Throws:

IOException

getS3AStreamStatistics

@InterfaceAudience.Private
 @InterfaceStability.Unstable
public S3AInstrumentation.InputStreamStatistics getS3AStreamStatistics()

Access the input stream statistics. This is for internal testing and may be removed without warning.

Returns:

the statistics for this input stream

setReadahead

public void setReadahead(Long readahead)

Specified by:

setReadahead in interface org.apache.hadoop.fs.CanSetReadahead

getReadahead

public long getReadahead()

Get the current readahead value.

Returns:

a non-negative readahead value

validateReadahead

public static long validateReadahead(@Nullable
                                     Long readahead)

from a possibly null Long value, return a valid readahead.

Parameters:

readahead - new readahead

Returns:

a natural number.

Throws:

IllegalArgumentException - if the range is invalid.

unbuffer

public void unbuffer()

Closes the underlying S3 stream, and merges the streamStatistics instance associated with the stream.

Specified by:

unbuffer in interface org.apache.hadoop.fs.CanUnbuffer

hasCapability

public boolean hasCapability(String capability)

Specified by:

hasCapability in interface org.apache.hadoop.fs.StreamCapabilities