org.apache.hadoop.fs.s3a

Class WriteOperationHelper

java.lang.Object

org.apache.hadoop.fs.s3a.WriteOperationHelper

All Implemented Interfaces:

Closeable, AutoCloseable, WriteOperations, org.apache.hadoop.fs.store.audit.AuditSpanSource

@InterfaceAudience.Private
 @InterfaceStability.Unstable
public class WriteOperationHelper
extends Object
implements WriteOperations

Helper for low-level operations against an S3 Bucket for writing data, creating and committing pending writes, and other S3-layer operations.

It hides direct access to the S3 API and is a location where the object operations can be evolved/enhanced.

Features

• Methods to create and submit requests to S3, so avoiding all direct interaction with the AWS APIs.
• Some extra preflight checks of arguments, so failing fast on errors.
• Callbacks to let the FS know of events in the output stream upload process.
• Other low-level access to S3 functions, for private use.
• Failure handling, including converting exceptions to IOEs.
• Integration with instrumentation.

This API is for internal use only. Span scoping: This helper is instantiated with span; it will be used before operations which query/update S3 History

 - A nested class in S3AFileSystem
 - Single shared instance created and reused.
 - [HADOOP-13786] A separate class, single instance in S3AFS
 - [HDFS-13934] Split into interface and implementation
 - [HADOOP-15711] Adds audit tracking; one instance per use.
 

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static interface	WriteOperationHelper.WriteOperationHelperCallbacks Callbacks for writeOperationHelper.

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	WriteOperationHelper(S3AFileSystem owner, org.apache.hadoop.conf.Configuration conf, S3AStatisticsContext statisticsContext, org.apache.hadoop.fs.store.audit.AuditSpanSource auditSpanSource, org.apache.hadoop.fs.store.audit.AuditSpan auditSpan, WriteOperationHelper.WriteOperationHelperCallbacks writeOperationHelperCallbacks) Constructor.

Method Summary

Modifier and Type	Method and Description
void	abortMultipartCommit(String destKey, String uploadId) Abort a multipart commit operation.
void	abortMultipartUpload(software.amazon.awssdk.services.s3.model.MultipartUpload upload) Abort a multipart commit operation.
void	abortMultipartUpload(String destKey, String uploadId, boolean shouldRetry, Invoker.Retried retrying) Abort a multipart upload operation.
int	abortMultipartUploadsUnderPath(String prefix) Abort multipart uploads under a path: limited to the first few hundred.
void	close() Deactivate the audit span.
software.amazon.awssdk.services.s3.model.CompleteMultipartUploadResponse	commitUpload(String destKey, String uploadId, List<software.amazon.awssdk.services.s3.model.CompletedPart> partETags, long length) This completes a multipart upload to the destination key via finalizeMultipartUpload().
software.amazon.awssdk.services.s3.model.CompleteMultipartUploadResponse	completeMPUwithRetries(String destKey, String uploadId, List<software.amazon.awssdk.services.s3.model.CompletedPart> partETags, long length, AtomicInteger errorCount, PutObjectOptions putOptions) This completes a multipart upload to the destination key via finalizeMultipartUpload().
software.amazon.awssdk.services.s3.model.PutObjectRequest	createPutObjectRequest(String destKey, long length, PutObjectOptions options, boolean isFile) Create a PutObjectRequest request against the specific key.
org.apache.hadoop.fs.store.audit.AuditSpan	createSpan(String operation, String path1, String path2)
org.apache.hadoop.fs.store.audit.AuditSpan	getAuditSpan() Get the audit span this object was created with.
org.apache.hadoop.conf.Configuration	getConf() Get the configuration of this instance; essentially the owning filesystem configuration.
RequestFactory	getRequestFactory() Get the request factory which uses this store's audit span.
void	incrementWriteOperations() Increment the write operation counter of the filesystem.
String	initiateMultiPartUpload(String destKey, PutObjectOptions options) Start the multipart upload process.
List<software.amazon.awssdk.services.s3.model.MultipartUpload>	listMultipartUploads(String prefix) List in-progress multipart uploads under a path: limited to the first few hundred.
software.amazon.awssdk.services.s3.model.UploadPartRequest.Builder	newUploadPartRequestBuilder(String destKey, String uploadId, int partNumber, long size) Create and initialize a part request builder of a multipart upload.
software.amazon.awssdk.services.s3.model.PutObjectResponse	putObject(software.amazon.awssdk.services.s3.model.PutObjectRequest putObjectRequest, PutObjectOptions putOptions, S3ADataBlocks.BlockUploadData uploadData, boolean isFile, org.apache.hadoop.fs.statistics.DurationTrackerFactory durationTrackerFactory) PUT an object directly (i.e.
<T> T	retry(String action, String path, boolean idempotent, org.apache.hadoop.util.functional.CallableRaisingIOE<T> operation) Execute a function with retry processing.
void	revertCommit(String destKey) Revert a commit by deleting the file.
String	toString() The toString method is intended to be used in logging/toString calls.
software.amazon.awssdk.services.s3.model.UploadPartResponse	uploadPart(software.amazon.awssdk.services.s3.model.UploadPartRequest request, software.amazon.awssdk.core.sync.RequestBody body, org.apache.hadoop.fs.statistics.DurationTrackerFactory durationTrackerFactory) Upload part of a multi-partition file.
void	writeFailed(Exception ex) Callback on a write failure.
void	writeSuccessful(long length) Callback on a successful write.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

WriteOperationHelper

protected WriteOperationHelper(S3AFileSystem owner,
                               org.apache.hadoop.conf.Configuration conf,
                               S3AStatisticsContext statisticsContext,
                               org.apache.hadoop.fs.store.audit.AuditSpanSource auditSpanSource,
                               org.apache.hadoop.fs.store.audit.AuditSpan auditSpan,
                               WriteOperationHelper.WriteOperationHelperCallbacks writeOperationHelperCallbacks)

Constructor.

Parameters:

owner - owner FS creating the helper

conf - Configuration object

statisticsContext - statistics context

auditSpanSource - source of spans

auditSpan - span to activate

writeOperationHelperCallbacks - callbacks used by writeOperationHelper

Method Detail

retry

public <T> T retry(String action,
                   String path,
                   boolean idempotent,
                   org.apache.hadoop.util.functional.CallableRaisingIOE<T> operation)
            throws IOException

Execute a function with retry processing. Also activates the current span.

Specified by:

retry in interface WriteOperations

Type Parameters:

T - type of return value

Parameters:

action - action to execute (used in error messages)

path - path of work (used in error messages)

idempotent - does the operation have semantics which mean that it can be retried even if was already executed?

operation - operation to execute

Returns:

the result of the call

Throws:

IOException - any IOE raised, or translated exception

getAuditSpan

public org.apache.hadoop.fs.store.audit.AuditSpan getAuditSpan()

Get the audit span this object was created with.

Returns:

the audit span

createPutObjectRequest

@Retries.OnceRaw
public software.amazon.awssdk.services.s3.model.PutObjectRequest createPutObjectRequest(String destKey,
                                                                                                         long length,
                                                                                                         PutObjectOptions options,
                                                                                                         boolean isFile)

Create a PutObjectRequest request against the specific key.

Specified by:

createPutObjectRequest in interface WriteOperations

Parameters:

destKey - destination key

length - size, if known. Use -1 for not known

options - options for the request

isFile - is data to be uploaded a file

Returns:

the request

writeSuccessful

public void writeSuccessful(long length)

Callback on a successful write.

Specified by:

writeSuccessful in interface WriteOperations

Parameters:

length - length of the write

writeFailed

public void writeFailed(Exception ex)

Callback on a write failure.

Specified by:

writeFailed in interface WriteOperations

Parameters:

ex - Any exception raised which triggered the failure.

initiateMultiPartUpload

@Retries.RetryTranslated
public String initiateMultiPartUpload(String destKey,
                                                               PutObjectOptions options)
                                                        throws IOException

Start the multipart upload process. Retry policy: retrying, translated.

Specified by:

initiateMultiPartUpload in interface WriteOperations

Parameters:

destKey - destination of upload

options - options for the put request

Returns:

the upload result containing the ID

Throws:

IOException - IO problem

completeMPUwithRetries

@Retries.RetryTranslated
public software.amazon.awssdk.services.s3.model.CompleteMultipartUploadResponse completeMPUwithRetries(String destKey,
                                                                                                                                String uploadId,
                                                                                                                                List<software.amazon.awssdk.services.s3.model.CompletedPart> partETags,
                                                                                                                                long length,
                                                                                                                                AtomicInteger errorCount,
                                                                                                                                PutObjectOptions putOptions)
                                                                                                                         throws IOException

This completes a multipart upload to the destination key via finalizeMultipartUpload(). Retry policy: retrying, translated. Retries increment the errorCount counter.

Specified by:

completeMPUwithRetries in interface WriteOperations

Parameters:

destKey - destination

uploadId - multipart operation Id

partETags - list of partial uploads

length - length of the upload

errorCount - a counter incremented by 1 on every error; for use in statistics

putOptions - put object options

Returns:

the result of the operation.

Throws:

IOException - if problems arose which could not be retried, or the retry count was exceeded

abortMultipartUpload

@Retries.RetryTranslated
public void abortMultipartUpload(String destKey,
                                                          String uploadId,
                                                          boolean shouldRetry,
                                                          Invoker.Retried retrying)
                                                   throws IOException

Abort a multipart upload operation.

Specified by:

abortMultipartUpload in interface WriteOperations

Parameters:

destKey - destination key of the upload

uploadId - multipart operation Id

shouldRetry - should failures trigger a retry?

retrying - callback invoked on every retry

Throws:

IOException - failure to abort

FileNotFoundException - if the abort ID is unknown

abortMultipartUpload

@Retries.RetryTranslated
public void abortMultipartUpload(software.amazon.awssdk.services.s3.model.MultipartUpload upload)
                                                   throws IOException

Abort a multipart commit operation.

Specified by:

abortMultipartUpload in interface WriteOperations

Parameters:

upload - upload to abort.

Throws:

IOException - on problems.

abortMultipartUploadsUnderPath

@Retries.RetryTranslated
public int abortMultipartUploadsUnderPath(String prefix)
                                                            throws IOException

Abort multipart uploads under a path: limited to the first few hundred.

Specified by:

abortMultipartUploadsUnderPath in interface WriteOperations

Parameters:

prefix - prefix for uploads to abort

Returns:

a count of aborts

Throws:

IOException - trouble; FileNotFoundExceptions are swallowed.

listMultipartUploads

@Retries.RetryTranslated
public List<software.amazon.awssdk.services.s3.model.MultipartUpload> listMultipartUploads(String prefix)
                                                                                                             throws IOException

Description copied from interface: WriteOperations

List in-progress multipart uploads under a path: limited to the first few hundred.

Specified by:

listMultipartUploads in interface WriteOperations

Parameters:

prefix - prefix for uploads to list

Returns:

a list of in-progress multipart uploads

Throws:

IOException - on problems

abortMultipartCommit

@Retries.RetryTranslated
public void abortMultipartCommit(String destKey,
                                                          String uploadId)
                                                   throws IOException

Abort a multipart commit operation.

Specified by:

abortMultipartCommit in interface WriteOperations

Parameters:

destKey - destination key of ongoing operation

uploadId - multipart operation Id

Throws:

IOException - on problems.

FileNotFoundException - if the abort ID is unknown

newUploadPartRequestBuilder

@Retries.OnceTranslated
public software.amazon.awssdk.services.s3.model.UploadPartRequest.Builder newUploadPartRequestBuilder(String destKey,
                                                                                                                              String uploadId,
                                                                                                                              int partNumber,
                                                                                                                              long size)
                                                                                                                       throws IOException

Create and initialize a part request builder of a multipart upload. The part number must be less than 10000. Retry policy is once-translated; to much effort

Specified by:

newUploadPartRequestBuilder in interface WriteOperations

Parameters:

destKey - destination key of ongoing operation

uploadId - ID of ongoing upload

partNumber - current part number of the upload

size - amount of data

Returns:

the request builder.

Throws:

IllegalArgumentException - if the parameters are invalid.

org.apache.hadoop.fs.PathIOException - if the part number is out of range.

IOException

toString

public String toString()

The toString method is intended to be used in logging/toString calls.

Overrides:

toString in class Object

Returns:

a string description.

putObject

@Retries.RetryTranslated
public software.amazon.awssdk.services.s3.model.PutObjectResponse putObject(software.amazon.awssdk.services.s3.model.PutObjectRequest putObjectRequest,
                                                                                                     PutObjectOptions putOptions,
                                                                                                     S3ADataBlocks.BlockUploadData uploadData,
                                                                                                     boolean isFile,
                                                                                                     org.apache.hadoop.fs.statistics.DurationTrackerFactory durationTrackerFactory)
                                                                                              throws IOException

PUT an object directly (i.e. not via the transfer manager). Byte length is calculated from the file length, or, if there is no file, from the content length of the header.

Specified by:

putObject in interface WriteOperations

Parameters:

putObjectRequest - the request

putOptions - put object options

durationTrackerFactory - factory for duration tracking

uploadData - data to be uploaded

isFile - is data to be uploaded a file

Returns:

the upload initiated

Throws:

IOException - on problems

revertCommit

@Retries.OnceTranslated
public void revertCommit(String destKey)
                                          throws IOException

Revert a commit by deleting the file. Relies on retry code in filesystem. Does not attempt to recreate the parent directory

Specified by:

revertCommit in interface WriteOperations

Parameters:

destKey - destination key

Throws:

IOException - on problems

commitUpload

@Retries.RetryTranslated
public software.amazon.awssdk.services.s3.model.CompleteMultipartUploadResponse commitUpload(String destKey,
                                                                                                                      String uploadId,
                                                                                                                      List<software.amazon.awssdk.services.s3.model.CompletedPart> partETags,
                                                                                                                      long length)
                                                                                                               throws IOException

This completes a multipart upload to the destination key via finalizeMultipartUpload(). Markers are never deleted on commit; this avoids having to issue many duplicate deletions. Retry policy: retrying, translated. Retries increment the errorCount counter.

Specified by:

commitUpload in interface WriteOperations

Parameters:

destKey - destination

uploadId - multipart operation Id

partETags - list of partial uploads

length - length of the upload

Returns:

the result of the operation.

Throws:

IOException - if problems arose which could not be retried, or the retry count was exceeded

uploadPart

@Retries.RetryTranslated
public software.amazon.awssdk.services.s3.model.UploadPartResponse uploadPart(software.amazon.awssdk.services.s3.model.UploadPartRequest request,
                                                                                                       software.amazon.awssdk.core.sync.RequestBody body,
                                                                                                       org.apache.hadoop.fs.statistics.DurationTrackerFactory durationTrackerFactory)
                                                                                                throws IOException

Upload part of a multi-partition file.

Specified by:

uploadPart in interface WriteOperations

Parameters:

request - request

durationTrackerFactory - duration tracker factory for operation

request - the upload part request.

body - the request body.

Returns:

the result of the operation.

Throws:

IOException - on problems

getConf

public org.apache.hadoop.conf.Configuration getConf()

Get the configuration of this instance; essentially the owning filesystem configuration.

Specified by:

getConf in interface WriteOperations

Returns:

the configuration.

createSpan

public org.apache.hadoop.fs.store.audit.AuditSpan createSpan(String operation,
                                                             @Nullable
                                                             String path1,
                                                             @Nullable
                                                             String path2)
                                                      throws IOException

Specified by:

createSpan in interface org.apache.hadoop.fs.store.audit.AuditSpanSource

Throws:

IOException

incrementWriteOperations

public void incrementWriteOperations()

Description copied from interface: WriteOperations

Increment the write operation counter of the filesystem.

Specified by:

incrementWriteOperations in interface WriteOperations

close

public void close()
           throws IOException

Deactivate the audit span.

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Throws:

IOException

getRequestFactory

public RequestFactory getRequestFactory()

Get the request factory which uses this store's audit span.

Returns:

the request factory.