public interface WriteOperations extends org.apache.hadoop.fs.store.audit.AuditSpanSource, Closeable
Modifier and Type | Method and Description |
---|---|
void |
abortMultipartCommit(String destKey,
String uploadId)
Abort a multipart commit operation.
|
void |
abortMultipartUpload(software.amazon.awssdk.services.s3.model.MultipartUpload upload)
Abort a multipart commit operation.
|
void |
abortMultipartUpload(String destKey,
String uploadId,
boolean shouldRetry,
Invoker.Retried retrying)
Abort a multipart upload operation.
|
int |
abortMultipartUploadsUnderPath(String prefix)
Abort multipart uploads under a path: limited to the first
few hundred.
|
software.amazon.awssdk.services.s3.model.CompleteMultipartUploadResponse |
commitUpload(String destKey,
String uploadId,
List<software.amazon.awssdk.services.s3.model.CompletedPart> partETags,
long length)
This completes a multipart upload to the destination key via
finalizeMultipartUpload() . |
software.amazon.awssdk.services.s3.model.CompleteMultipartUploadResponse |
completeMPUwithRetries(String destKey,
String uploadId,
List<software.amazon.awssdk.services.s3.model.CompletedPart> partETags,
long length,
AtomicInteger errorCount,
PutObjectOptions putOptions)
This completes a multipart upload to the destination key via
finalizeMultipartUpload() . |
software.amazon.awssdk.services.s3.model.PutObjectRequest |
createPutObjectRequest(String destKey,
long length,
PutObjectOptions options,
boolean isFile)
Create a
PutObjectRequest request against the specific key. |
org.apache.hadoop.conf.Configuration |
getConf()
Get the configuration of this instance; essentially the owning
filesystem configuration.
|
void |
incrementWriteOperations()
Increment the write operation counter
of the filesystem.
|
String |
initiateMultiPartUpload(String destKey,
PutObjectOptions options)
Start the multipart upload process.
|
List<software.amazon.awssdk.services.s3.model.MultipartUpload> |
listMultipartUploads(String prefix)
List in-progress multipart uploads under a path: limited to the first
few hundred.
|
software.amazon.awssdk.services.s3.model.UploadPartRequest.Builder |
newUploadPartRequestBuilder(String destKey,
String uploadId,
int partNumber,
long size)
Create and initialize a part request builder of a multipart upload.
|
software.amazon.awssdk.services.s3.model.PutObjectResponse |
putObject(software.amazon.awssdk.services.s3.model.PutObjectRequest putObjectRequest,
PutObjectOptions putOptions,
S3ADataBlocks.BlockUploadData uploadData,
boolean isFile,
org.apache.hadoop.fs.statistics.DurationTrackerFactory durationTrackerFactory)
PUT an object directly (i.e.
|
<T> T |
retry(String action,
String path,
boolean idempotent,
org.apache.hadoop.util.functional.CallableRaisingIOE<T> operation)
Execute a function with retry processing.
|
void |
revertCommit(String destKey)
Revert a commit by deleting the file.
|
software.amazon.awssdk.services.s3.model.UploadPartResponse |
uploadPart(software.amazon.awssdk.services.s3.model.UploadPartRequest request,
software.amazon.awssdk.core.sync.RequestBody body,
org.apache.hadoop.fs.statistics.DurationTrackerFactory durationTrackerFactory)
Upload part of a multi-partition file.
|
void |
writeFailed(Exception ex)
Callback on a write failure.
|
void |
writeSuccessful(long length)
Callback on a successful write.
|
<T> T retry(String action, String path, boolean idempotent, org.apache.hadoop.util.functional.CallableRaisingIOE<T> operation) throws IOException
T
- type of return valueaction
- action to execute (used in error messages)path
- path of work (used in error messages)idempotent
- does the operation have semantics
which mean that it can be retried even if was already executed?operation
- operation to executeIOException
- any IOE raised, or translated exceptionsoftware.amazon.awssdk.services.s3.model.PutObjectRequest createPutObjectRequest(String destKey, long length, @Nullable PutObjectOptions options, boolean isFile)
PutObjectRequest
request against the specific key.destKey
- destination keylength
- size, if known. Use -1 for not knownoptions
- options for the requestisFile
- is data to be uploaded a filevoid writeSuccessful(long length)
length
- length of the writevoid writeFailed(Exception ex)
ex
- Any exception raised which triggered the failure.@Retries.RetryTranslated String initiateMultiPartUpload(String destKey, PutObjectOptions options) throws IOException
destKey
- destination of uploadoptions
- options for the put requestIOException
- IO problem@Retries.RetryTranslated software.amazon.awssdk.services.s3.model.CompleteMultipartUploadResponse completeMPUwithRetries(String destKey, String uploadId, List<software.amazon.awssdk.services.s3.model.CompletedPart> partETags, long length, AtomicInteger errorCount, PutObjectOptions putOptions) throws IOException
finalizeMultipartUpload()
.
Retry policy: retrying, translated.
Retries increment the errorCount
counter.destKey
- destinationuploadId
- multipart operation IdpartETags
- list of partial uploadslength
- length of the uploaderrorCount
- a counter incremented by 1 on every error; for
use in statisticsputOptions
- put object optionsIOException
- if problems arose which could not be retried, or
the retry count was exceeded@Retries.RetryTranslated void abortMultipartUpload(String destKey, String uploadId, boolean shouldRetry, Invoker.Retried retrying) throws IOException
destKey
- destination key of the uploaduploadId
- multipart operation IdshouldRetry
- should failures trigger a retry?retrying
- callback invoked on every retryIOException
- failure to abortFileNotFoundException
- if the abort ID is unknown@Retries.RetryTranslated void abortMultipartUpload(software.amazon.awssdk.services.s3.model.MultipartUpload upload) throws IOException
upload
- upload to abort.IOException
- on problems.@Retries.RetryTranslated int abortMultipartUploadsUnderPath(String prefix) throws IOException
prefix
- prefix for uploads to abortIOException
- trouble; FileNotFoundExceptions are swallowed.List<software.amazon.awssdk.services.s3.model.MultipartUpload> listMultipartUploads(String prefix) throws IOException
prefix
- prefix for uploads to listIOException
- on problems@Retries.RetryTranslated void abortMultipartCommit(String destKey, String uploadId) throws IOException
destKey
- destination key of ongoing operationuploadId
- multipart operation IdIOException
- on problems.FileNotFoundException
- if the abort ID is unknownsoftware.amazon.awssdk.services.s3.model.UploadPartRequest.Builder newUploadPartRequestBuilder(String destKey, String uploadId, int partNumber, long size) throws IOException
destKey
- destination key of ongoing operationuploadId
- ID of ongoing uploadpartNumber
- current part number of the uploadsize
- amount of dataIllegalArgumentException
- if the parameters are invalidorg.apache.hadoop.fs.PathIOException
- if the part number is out of range.IOException
@Retries.RetryTranslated software.amazon.awssdk.services.s3.model.PutObjectResponse putObject(software.amazon.awssdk.services.s3.model.PutObjectRequest putObjectRequest, PutObjectOptions putOptions, S3ADataBlocks.BlockUploadData uploadData, boolean isFile, org.apache.hadoop.fs.statistics.DurationTrackerFactory durationTrackerFactory) throws IOException
putObjectRequest
- the requestputOptions
- put object optionsdurationTrackerFactory
- factory for duration trackinguploadData
- data to be uploadedisFile
- is data to be uploaded a fileIOException
- on problems@Retries.OnceTranslated void revertCommit(String destKey) throws IOException
destKey
- destination keyIOException
- on problems@Retries.RetryTranslated software.amazon.awssdk.services.s3.model.CompleteMultipartUploadResponse commitUpload(String destKey, String uploadId, List<software.amazon.awssdk.services.s3.model.CompletedPart> partETags, long length) throws IOException
finalizeMultipartUpload()
.
Retry policy: retrying, translated.
Retries increment the errorCount
counter.destKey
- destinationuploadId
- multipart operation IdpartETags
- list of partial uploadslength
- length of the uploadIOException
- if problems arose which could not be retried, or
the retry count was exceeded@Retries.RetryTranslated software.amazon.awssdk.services.s3.model.UploadPartResponse uploadPart(software.amazon.awssdk.services.s3.model.UploadPartRequest request, software.amazon.awssdk.core.sync.RequestBody body, org.apache.hadoop.fs.statistics.DurationTrackerFactory durationTrackerFactory) throws IOException
request
- the upload part request.body
- the request body.durationTrackerFactory
- factory for duration trackingIOException
- on problemsorg.apache.hadoop.conf.Configuration getConf()
void incrementWriteOperations()
Copyright © 2008–2024 Apache Software Foundation. All rights reserved.