org.apache.hadoop.fs.s3a

Class WriteOperationHelper

java.lang.Object

org.apache.hadoop.fs.s3a.WriteOperationHelper

@InterfaceAudience.Private
 @InterfaceStability.Unstable
public class WriteOperationHelper
extends Object

Helper for low-level operations against an S3 Bucket for writing data, creating and committing pending writes, and other S3-layer operations.

It hides direct access to the S3 API and is a location where the object operations can be evolved/enhanced.

Features

• Methods to create and submit requests to S3, so avoiding all direct interaction with the AWS APIs.
• Some extra preflight checks of arguments, so failing fast on errors.
• Callbacks to let the FS know of events in the output stream upload process.
• Other low-level access to S3 functions, for private use.
• Failure handling, including converting exceptions to IOEs.
• Integration with instrumentation and S3Guard.
• Evolution to add more low-level operations, such as S3 select.

This API is for internal use only.

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	WriteOperationHelper(S3AFileSystem owner, org.apache.hadoop.conf.Configuration conf) Constructor.

Method Summary

Modifier and Type	Method and Description
void	abortMultipartCommit(String destKey, String uploadId) Abort a multipart commit operation.
void	abortMultipartUpload(com.amazonaws.services.s3.model.MultipartUpload upload) Abort a multipart commit operation.
void	abortMultipartUpload(String destKey, String uploadId, Invoker.Retried retrying) Abort a multipart upload operation.
int	abortMultipartUploadsUnderPath(String prefix) Abort multipart uploads under a path: limited to the first few hundred.
com.amazonaws.services.s3.model.CompleteMultipartUploadResult	commitUpload(String destKey, String uploadId, List<com.amazonaws.services.s3.model.PartETag> partETags, long length, BulkOperationState operationState) This completes a multipart upload to the destination key via finalizeMultipartUpload().
com.amazonaws.services.s3.model.CompleteMultipartUploadResult	completeMPUwithRetries(String destKey, String uploadId, List<com.amazonaws.services.s3.model.PartETag> partETags, long length, AtomicInteger errorCount) This completes a multipart upload to the destination key via finalizeMultipartUpload().
com.amazonaws.services.s3.model.PutObjectRequest	createPutObjectRequest(String dest, File sourceFile) Create a PutObjectRequest request to upload a file.
com.amazonaws.services.s3.model.PutObjectRequest	createPutObjectRequest(String destKey, InputStream inputStream, long length) Create a PutObjectRequest request against the specific key.
org.apache.hadoop.conf.Configuration	getConf() Get the configuration of this instance; essentially the owning filesystem configuration.
BulkOperationState	initiateCommitOperation(org.apache.hadoop.fs.Path path) Initiate a commit operation through any metastore.
String	initiateMultiPartUpload(String destKey) Start the multipart upload process.
com.amazonaws.services.s3.model.ObjectMetadata	newObjectMetadata(long length) Create a new object metadata instance.
com.amazonaws.services.s3.model.SelectObjectContentRequest	newSelectRequest(org.apache.hadoop.fs.Path path) Create a S3 Select request for the destination path.
com.amazonaws.services.s3.model.UploadPartRequest	newUploadPartRequest(String destKey, String uploadId, int partNumber, int size, InputStream uploadStream, File sourceFile, Long offset) Create and initialize a part request of a multipart upload.
com.amazonaws.services.s3.model.PutObjectResult	putObject(com.amazonaws.services.s3.model.PutObjectRequest putObjectRequest) PUT an object directly (i.e.
<T> T	retry(String action, String path, boolean idempotent, Invoker.Operation<T> operation) Execute a function with retry processing.
void	revertCommit(String destKey, BulkOperationState operationState) Revert a commit by deleting the file.
com.amazonaws.services.s3.model.SelectObjectContentResult	select(org.apache.hadoop.fs.Path source, com.amazonaws.services.s3.model.SelectObjectContentRequest request, String action) Execute an S3 Select operation.
String	toString() The toString method is intended to be used in logging/toString calls.
com.amazonaws.services.s3.transfer.model.UploadResult	uploadObject(com.amazonaws.services.s3.model.PutObjectRequest putObjectRequest) PUT an object via the transfer manager.
com.amazonaws.services.s3.model.UploadPartResult	uploadPart(com.amazonaws.services.s3.model.UploadPartRequest request) Upload part of a multi-partition file.
void	writeFailed(Exception ex) Callback on a write failure.
void	writeSuccessful(long length) Callback on a successful write.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

WriteOperationHelper

protected WriteOperationHelper(S3AFileSystem owner,
                               org.apache.hadoop.conf.Configuration conf)

Constructor.

Parameters:

owner - owner FS creating the helper

conf - Configuration object

Method Detail

retry

public <T> T retry(String action,
                   String path,
                   boolean idempotent,
                   Invoker.Operation<T> operation)
            throws IOException

Execute a function with retry processing.

Type Parameters:

T - type of return value

Parameters:

action - action to execute (used in error messages)

path - path of work (used in error messages)

idempotent - does the operation have semantics which mean that it can be retried even if was already executed?

operation - operation to execute

Returns:

the result of the call

Throws:

IOException - any IOE raised, or translated exception

createPutObjectRequest

public com.amazonaws.services.s3.model.PutObjectRequest createPutObjectRequest(String destKey,
                                                                               InputStream inputStream,
                                                                               long length)

Create a PutObjectRequest request against the specific key.

Parameters:

destKey - destination key

inputStream - source data.

length - size, if known. Use -1 for not known

Returns:

the request

createPutObjectRequest

public com.amazonaws.services.s3.model.PutObjectRequest createPutObjectRequest(String dest,
                                                                               File sourceFile)

Create a PutObjectRequest request to upload a file.

Parameters:

dest - key to PUT to.

sourceFile - source file

Returns:

the request

writeSuccessful

public void writeSuccessful(long length)

Callback on a successful write.

Parameters:

length - length of the write

writeFailed

public void writeFailed(Exception ex)

Callback on a write failure.

Parameters:

ex - Any exception raised which triggered the failure.

newObjectMetadata

public com.amazonaws.services.s3.model.ObjectMetadata newObjectMetadata(long length)

Create a new object metadata instance. Any standard metadata headers are added here, for example: encryption.

Parameters:

length - size, if known. Use -1 for not known

Returns:

a new metadata instance

initiateMultiPartUpload

@Retries.RetryTranslated
public String initiateMultiPartUpload(String destKey)
                                                        throws IOException

Start the multipart upload process. Retry policy: retrying, translated.

Parameters:

destKey - destination of upload

Returns:

the upload result containing the ID

Throws:

IOException - IO problem

completeMPUwithRetries

@Retries.RetryTranslated
public com.amazonaws.services.s3.model.CompleteMultipartUploadResult completeMPUwithRetries(String destKey,
                                                                                                                     String uploadId,
                                                                                                                     List<com.amazonaws.services.s3.model.PartETag> partETags,
                                                                                                                     long length,
                                                                                                                     AtomicInteger errorCount)
                                                                                                              throws IOException

This completes a multipart upload to the destination key via finalizeMultipartUpload(). Retry policy: retrying, translated. Retries increment the errorCount counter.

Parameters:

destKey - destination

uploadId - multipart operation Id

partETags - list of partial uploads

length - length of the upload

errorCount - a counter incremented by 1 on every error; for use in statistics

Returns:

the result of the operation.

Throws:

IOException - if problems arose which could not be retried, or the retry count was exceeded

abortMultipartUpload

@Retries.RetryTranslated
public void abortMultipartUpload(String destKey,
                                                          String uploadId,
                                                          Invoker.Retried retrying)
                                                   throws IOException

Abort a multipart upload operation.

Parameters:

destKey - destination key of the upload

uploadId - multipart operation Id

retrying - callback invoked on every retry

Throws:

IOException - failure to abort

FileNotFoundException - if the abort ID is unknown

abortMultipartUpload

@Retries.RetryTranslated
public void abortMultipartUpload(com.amazonaws.services.s3.model.MultipartUpload upload)
                                                   throws IOException

Abort a multipart commit operation.

Parameters:

upload - upload to abort.

Throws:

IOException - on problems.

abortMultipartUploadsUnderPath

@Retries.RetryTranslated
public int abortMultipartUploadsUnderPath(String prefix)
                                                            throws IOException

Abort multipart uploads under a path: limited to the first few hundred.

Parameters:

prefix - prefix for uploads to abort

Returns:

a count of aborts

Throws:

IOException - trouble; FileNotFoundExceptions are swallowed.

abortMultipartCommit

@Retries.RetryTranslated
public void abortMultipartCommit(String destKey,
                                                          String uploadId)
                                                   throws IOException

Abort a multipart commit operation.

Parameters:

destKey - destination key of ongoing operation

uploadId - multipart operation Id

Throws:

IOException - on problems.

FileNotFoundException - if the abort ID is unknown

newUploadPartRequest

public com.amazonaws.services.s3.model.UploadPartRequest newUploadPartRequest(String destKey,
                                                                              String uploadId,
                                                                              int partNumber,
                                                                              int size,
                                                                              InputStream uploadStream,
                                                                              File sourceFile,
                                                                              Long offset)

Create and initialize a part request of a multipart upload. Exactly one of: uploadStream or sourceFile must be specified. A subset of the file may be posted, by providing the starting point in offset and a length of block in size equal to or less than the remaining bytes.

Parameters:

destKey - destination key of ongoing operation

uploadId - ID of ongoing upload

partNumber - current part number of the upload

size - amount of data

uploadStream - source of data to upload

sourceFile - optional source file.

offset - offset in file to start reading.

Returns:

the request.

toString

public String toString()

The toString method is intended to be used in logging/toString calls.

Overrides:

toString in class Object

Returns:

a string description.

putObject

@Retries.RetryTranslated
public com.amazonaws.services.s3.model.PutObjectResult putObject(com.amazonaws.services.s3.model.PutObjectRequest putObjectRequest)
                                                                                   throws IOException

PUT an object directly (i.e. not via the transfer manager). Byte length is calculated from the file length, or, if there is no file, from the content length of the header.

Parameters:

putObjectRequest - the request

Returns:

the upload initiated

Throws:

IOException - on problems

uploadObject

@Retries.RetryTranslated
public com.amazonaws.services.s3.transfer.model.UploadResult uploadObject(com.amazonaws.services.s3.model.PutObjectRequest putObjectRequest)
                                                                                            throws IOException

PUT an object via the transfer manager.

Parameters:

putObjectRequest - the request

Returns:

the result of the operation

Throws:

IOException - on problems

revertCommit

@Retries.OnceTranslated
public void revertCommit(String destKey,
                                                 @Nullable
                                                 BulkOperationState operationState)
                                          throws IOException

Revert a commit by deleting the file. Relies on retry code in filesystem

Parameters:

destKey - destination key

operationState - operational state for a bulk update

Throws:

IOException - on problems

commitUpload

@Retries.RetryTranslated
public com.amazonaws.services.s3.model.CompleteMultipartUploadResult commitUpload(String destKey,
                                                                                                           String uploadId,
                                                                                                           List<com.amazonaws.services.s3.model.PartETag> partETags,
                                                                                                           long length,
                                                                                                           @Nullable
                                                                                                           BulkOperationState operationState)
                                                                                                    throws IOException

This completes a multipart upload to the destination key via finalizeMultipartUpload(). Retry policy: retrying, translated. Retries increment the errorCount counter.

Parameters:

destKey - destination

uploadId - multipart operation Id

partETags - list of partial uploads

length - length of the upload

operationState - operational state for a bulk update

Returns:

the result of the operation.

Throws:

IOException - if problems arose which could not be retried, or the retry count was exceeded

initiateCommitOperation

public BulkOperationState initiateCommitOperation(org.apache.hadoop.fs.Path path)
                                           throws IOException

Initiate a commit operation through any metastore.

Parameters:

path - path under which the writes will all take place.

Returns:

an possibly null operation state from the metastore.

Throws:

IOException - failure to instantiate.

uploadPart

@Retries.RetryTranslated
public com.amazonaws.services.s3.model.UploadPartResult uploadPart(com.amazonaws.services.s3.model.UploadPartRequest request)
                                                                                     throws IOException

Upload part of a multi-partition file.

Parameters:

request - request

Returns:

the result of the operation.

Throws:

IOException - on problems

getConf

public org.apache.hadoop.conf.Configuration getConf()

Get the configuration of this instance; essentially the owning filesystem configuration.

Returns:

the configuration.

newSelectRequest

public com.amazonaws.services.s3.model.SelectObjectContentRequest newSelectRequest(org.apache.hadoop.fs.Path path)

Create a S3 Select request for the destination path. This does not build the query.

Parameters:

path - pre-qualified path for query

Returns:

the request

select

@Retries.RetryTranslated
public com.amazonaws.services.s3.model.SelectObjectContentResult select(org.apache.hadoop.fs.Path source,
                                                                                                 com.amazonaws.services.s3.model.SelectObjectContentRequest request,
                                                                                                 String action)
                                                                                          throws IOException

Execute an S3 Select operation. On a failure, the request is only logged at debug to avoid the select exception being printed.

Parameters:

source - source for selection

request - Select request to issue.

action - the action for use in exception creation

Returns:

response

Throws:

IOException - failure