Interface S3TransferManager
-
- All Superinterfaces:
AutoCloseable
,SdkAutoCloseable
@ThreadSafe public interface S3TransferManager extends SdkAutoCloseable
The S3 Transfer Manager offers a simple API that allows you to transfer a single object or a set of objects to and from Amazon S3 with enhanced throughput and reliability. It leverages Amazon S3 multipart upload and byte-range fetches to perform transfers in parallel. In addition, the S3 Transfer Manager also enables you to monitor a transfer's progress in real-time, as well as pause the transfer for execution at a later time.Instantiate the S3 Transfer Manager
Create a transfer manager instance with SDK default settings. Note that it's recommended to addsoftware.amazon.awssdk.crt:aws-crt
dependency so that automatic multipart feature can be enabled Create an S3 Transfer Manager instance with custom settingsCommon Usage Patterns
Upload a file to S3 Download an S3 object to a local file Upload a local directory to an S3 bucket Download S3 objects to a local directory Copy an S3 object to a different location in S3
-
-
Nested Class Summary
Nested Classes Modifier and Type Interface Description static interface
S3TransferManager.Builder
The builder definition for aS3TransferManager
.
-
Method Summary
All Methods Static Methods Instance Methods Default Methods Modifier and Type Method Description static S3TransferManager.Builder
builder()
Creates a default builder forS3TransferManager
.default Copy
copy(Consumer<CopyRequest.Builder> copyRequestBuilder)
This is a convenience method that creates an instance of theCopyRequest
builder, avoiding the need to create one manually viaCopyRequest.builder()
.default Copy
copy(CopyRequest copyRequest)
Creates a copy of an object that is already stored in S3.static S3TransferManager
create()
Create anS3TransferManager
using the default values.default <ResultT> Download<ResultT>
download(DownloadRequest<ResultT> downloadRequest)
Downloads an object identified by the bucket and key from S3 through the givenAsyncResponseTransformer
.default DirectoryDownload
downloadDirectory(Consumer<DownloadDirectoryRequest.Builder> requestBuilder)
This is a convenience method that creates an instance of theDownloadDirectoryRequest
builder, avoiding the need to create one manually viaDownloadDirectoryRequest.builder()
.default DirectoryDownload
downloadDirectory(DownloadDirectoryRequest downloadDirectoryRequest)
Downloads all objects under a bucket to the provided directory.default FileDownload
downloadFile(Consumer<DownloadFileRequest.Builder> request)
This is a convenience method that creates an instance of theDownloadFileRequest
builder, avoiding the need to create one manually viaDownloadFileRequest.builder()
.default FileDownload
downloadFile(DownloadFileRequest downloadRequest)
Downloads an object identified by the bucket and key from S3 to a local file.default FileDownload
resumeDownloadFile(Consumer<ResumableFileDownload.Builder> resumableFileDownload)
This is a convenience method that creates an instance of theResumableFileDownload
builder, avoiding the need to create one manually viaResumableFileDownload.builder()
.default FileDownload
resumeDownloadFile(ResumableFileDownload resumableFileDownload)
Resumes a downloadFile operation.default FileUpload
resumeUploadFile(Consumer<ResumableFileUpload.Builder> resumableFileUpload)
This is a convenience method that creates an instance of theResumableFileUpload
builder, avoiding the need to create one manually viaResumableFileUpload.builder()
.default FileUpload
resumeUploadFile(ResumableFileUpload resumableFileUpload)
Resumes uploadFile operation.default Upload
upload(Consumer<UploadRequest.Builder> request)
This is a convenience method that creates an instance of theUploadRequest
builder, avoiding the need to create one manually viaUploadRequest.builder()
.default Upload
upload(UploadRequest uploadRequest)
Uploads the givenAsyncRequestBody
to an object in S3.default DirectoryUpload
uploadDirectory(Consumer<UploadDirectoryRequest.Builder> requestBuilder)
This is a convenience method that creates an instance of theUploadDirectoryRequest
builder, avoiding the need to create one manually viaUploadDirectoryRequest.builder()
.default DirectoryUpload
uploadDirectory(UploadDirectoryRequest uploadDirectoryRequest)
Uploads all files under the given directory to the provided S3 bucket.default FileUpload
uploadFile(Consumer<UploadFileRequest.Builder> request)
This is a convenience method that creates an instance of theUploadFileRequest
builder, avoiding the need to create one manually viaUploadFileRequest.builder()
.default FileUpload
uploadFile(UploadFileRequest uploadFileRequest)
Uploads a local file to an object in S3.-
Methods inherited from interface software.amazon.awssdk.utils.SdkAutoCloseable
close
-
-
-
-
Method Detail
-
downloadFile
default FileDownload downloadFile(DownloadFileRequest downloadRequest)
Downloads an object identified by the bucket and key from S3 to a local file. For non-file-based downloads, you may usedownload(DownloadRequest)
instead.The SDK will create a new file if the provided one doesn't exist. The default permission for the new file depends on the file system and platform. Users can configure the permission on the file using Java API by themselves. If the file already exists, the SDK will replace it. In the event of an error, the SDK will NOT attempt to delete the file, leaving it as-is.
Users can monitor the progress of the transfer by attaching a
TransferListener
. The providedLoggingTransferListener
logs a basic progress bar; users can also implement their own listeners.Usage Example:
- See Also:
downloadFile(Consumer)
,download(DownloadRequest)
-
downloadFile
default FileDownload downloadFile(Consumer<DownloadFileRequest.Builder> request)
This is a convenience method that creates an instance of theDownloadFileRequest
builder, avoiding the need to create one manually viaDownloadFileRequest.builder()
.- See Also:
downloadFile(DownloadFileRequest)
-
resumeDownloadFile
default FileDownload resumeDownloadFile(ResumableFileDownload resumableFileDownload)
Resumes a downloadFile operation. This download operation uses the same configuration as the original download. Any content that has already been fetched since the last pause will be skipped and only the remaining data will be downloaded from Amazon S3.If it is determined that the source S3 object or the destination file has be modified since the last pause, the SDK will download the object from the beginning as if it is a new
DownloadFileRequest
.Usage Example:
- Parameters:
resumableFileDownload
- the download to resume.- Returns:
- A new
FileDownload
object to use to check the state of the download. - See Also:
downloadFile(DownloadFileRequest)
,FileDownload.pause()
-
resumeDownloadFile
default FileDownload resumeDownloadFile(Consumer<ResumableFileDownload.Builder> resumableFileDownload)
This is a convenience method that creates an instance of theResumableFileDownload
builder, avoiding the need to create one manually viaResumableFileDownload.builder()
.
-
download
default <ResultT> Download<ResultT> download(DownloadRequest<ResultT> downloadRequest)
Downloads an object identified by the bucket and key from S3 through the givenAsyncResponseTransformer
. For downloading to a file, you may usedownloadFile(DownloadFileRequest)
instead.Users can monitor the progress of the transfer by attaching a
TransferListener
. The providedLoggingTransferListener
logs a basic progress bar; users can also implement their own listeners.Usage Example (this example buffers the entire object in memory and is not suitable for large objects):
See the static factory methods available in
AsyncResponseTransformer
for other use cases.- Type Parameters:
ResultT
- The type of data theAsyncResponseTransformer
produces- Parameters:
downloadRequest
- the download request, containing aGetObjectRequest
andAsyncResponseTransformer
- Returns:
- A
Download
that can be used to track the ongoing transfer - See Also:
downloadFile(DownloadFileRequest)
-
uploadFile
default FileUpload uploadFile(UploadFileRequest uploadFileRequest)
Uploads a local file to an object in S3. For non-file-based uploads, you may useupload(UploadRequest)
instead.Users can monitor the progress of the transfer by attaching a
TransferListener
. The providedLoggingTransferListener
logs a basic progress bar; users can also implement their own listeners. Upload a local file to an object in S3. For non-file-based uploads, you may useupload(UploadRequest)
instead.Usage Example:
- See Also:
uploadFile(Consumer)
,upload(UploadRequest)
-
uploadFile
default FileUpload uploadFile(Consumer<UploadFileRequest.Builder> request)
This is a convenience method that creates an instance of theUploadFileRequest
builder, avoiding the need to create one manually viaUploadFileRequest.builder()
.- See Also:
uploadFile(UploadFileRequest)
-
resumeUploadFile
default FileUpload resumeUploadFile(ResumableFileUpload resumableFileUpload)
Resumes uploadFile operation. This upload operation will use the same configuration provided inResumableFileUpload
. The SDK will skip the data that has already been upload since the last pause and only upload the remaining data from the source file.If it is determined that the source file has be modified since the last pause, the SDK will upload the object from the beginning as if it is a new
UploadFileRequest
.Usage Example:
- Parameters:
resumableFileUpload
- the upload to resume.- Returns:
- A new
FileUpload
object to use to check the state of the download. - See Also:
uploadFile(UploadFileRequest)
,FileUpload.pause()
-
resumeUploadFile
default FileUpload resumeUploadFile(Consumer<ResumableFileUpload.Builder> resumableFileUpload)
This is a convenience method that creates an instance of theResumableFileUpload
builder, avoiding the need to create one manually viaResumableFileUpload.builder()
.- See Also:
resumeUploadFile(ResumableFileUpload)
-
upload
default Upload upload(UploadRequest uploadRequest)
Uploads the givenAsyncRequestBody
to an object in S3. For file-based uploads, you may useuploadFile(UploadFileRequest)
instead.Users can monitor the progress of the transfer by attaching a
TransferListener
. The providedLoggingTransferListener
logs a basic progress bar; users can also implement their own listeners.Usage Example: See the static factory methods available in
AsyncRequestBody
for other use cases.- Parameters:
uploadRequest
- the upload request, containing aPutObjectRequest
andAsyncRequestBody
- Returns:
- An
Upload
that can be used to track the ongoing transfer - See Also:
upload(Consumer)
,uploadFile(UploadFileRequest)
-
upload
default Upload upload(Consumer<UploadRequest.Builder> request)
This is a convenience method that creates an instance of theUploadRequest
builder, avoiding the need to create one manually viaUploadRequest.builder()
.- See Also:
upload(UploadRequest)
-
uploadDirectory
default DirectoryUpload uploadDirectory(UploadDirectoryRequest uploadDirectoryRequest)
Uploads all files under the given directory to the provided S3 bucket. The key name transformation depends on the optional prefix and delimiter provided in theUploadDirectoryRequest
. By default, all subdirectories will be uploaded recursively, and symbolic links are not followed automatically. This behavior can be configured in at request level viaUploadDirectoryRequest.Builder.followSymbolicLinks(Boolean)
or client level viaS3TransferManager.Builder.uploadDirectoryFollowSymbolicLinks(Boolean)
Note that request-level configuration takes precedence over client-level configuration.By default, the prefix is an empty string and the delimiter is
"/"
. Assume you have a local directory "/test" with the following structure:|- test |- sample.jpg |- photos |- 2022 |- January |- sample.jpg |- February |- sample1.jpg |- sample2.jpg |- sample3.jpg
- sample.jpg
- photos/2022/January/sample.jpg
- photos/2022/February/sample1.jpg
- photos/2022/February/sample2.jpg
- photos/2022/February/sample3.jpg
The returned
CompletableFuture
only completes exceptionally if the request cannot be attempted as a whole (the source directory provided does not exist for example). The future completes successfully for partial successful requests, i.e., there might be failed uploads in the successfully completed response. As a result, you should check for errors in the response viaCompletedDirectoryUpload.failedTransfers()
even when the future completes successfully.The current user must have read access to all directories and files.
Usage Example:
- Parameters:
uploadDirectoryRequest
- the upload directory request- See Also:
uploadDirectory(Consumer)
-
uploadDirectory
default DirectoryUpload uploadDirectory(Consumer<UploadDirectoryRequest.Builder> requestBuilder)
This is a convenience method that creates an instance of theUploadDirectoryRequest
builder, avoiding the need to create one manually viaUploadDirectoryRequest.builder()
.- See Also:
uploadDirectory(UploadDirectoryRequest)
-
downloadDirectory
default DirectoryDownload downloadDirectory(DownloadDirectoryRequest downloadDirectoryRequest)
Downloads all objects under a bucket to the provided directory. By default, all objects in the entire bucket will be downloaded. You can modify this behavior by providing aDownloadDirectoryRequest.listObjectsRequestTransformer()
and/or aDownloadDirectoryRequest.filter()
inDownloadDirectoryRequest
to limit the S3 objects to download.The downloaded directory structure will match with the provided S3 virtual bucket. For example, assume that you have the following keys in your bucket:
- sample.jpg
- photos/2022/January/sample.jpg
- photos/2022/February/sample1.jpg
- photos/2022/February/sample2.jpg
- photos/2022/February/sample3.jpg
|- test |- sample.jpg |- photos |- 2022 |- January |- sample.jpg |- February |- sample1.jpg |- sample2.jpg |- sample3.jpg
The returned
CompletableFuture
only completes exceptionally if the request cannot be attempted as a whole (the downloadDirectoryRequest is invalid for example). The future completes successfully for partial successful requests, i.e., there might be failed downloads in a successfully completed response. As a result, you should check for errors in the response viaCompletedDirectoryDownload.failedTransfers()
even when the future completes successfully.The SDK will create the destination directory if it does not already exist. If a specific file already exists, the existing content will be replaced with the corresponding S3 object content.
The current user must have write access to all directories and files
Usage Example:
- Parameters:
downloadDirectoryRequest
- the download directory request- See Also:
downloadDirectory(Consumer)
-
downloadDirectory
default DirectoryDownload downloadDirectory(Consumer<DownloadDirectoryRequest.Builder> requestBuilder)
This is a convenience method that creates an instance of theDownloadDirectoryRequest
builder, avoiding the need to create one manually viaDownloadDirectoryRequest.builder()
.
-
copy
default Copy copy(CopyRequest copyRequest)
Creates a copy of an object that is already stored in S3.Depending on the underlying S3Client,
S3TransferManager
may intelligently use plainCopyObjectRequest
s for smaller objects, and multiple parallelUploadPartCopyRequest
s for larger objects. If multipart copy is supported by the underlying S3Client, this behavior can be configured viaS3CrtAsyncClientBuilder.minimumPartSizeInBytes(Long)
. Note that for multipart copy request, existing metadata stored in the source object is NOT copied to the destination object; if required, you can retrieve the metadata from the source object and set it explicitly in the @link CopyObjectRequest.Builder#metadata(Map)}.While this API supports
TransferListener
s, they will not receivebytesTransferred
callback-updates due to the way theCopyObjectRequest
API behaves. When copying an object, S3 performs the byte copying on your behalf while keeping the connection alive. The progress of the copy is not known until it fully completes and S3 sends a response describing the outcome.If you are copying an object to a bucket in a different region, you need to enable cross region access on the
S3AsyncClient
.Usage Example:
- Parameters:
copyRequest
- the copy request, containing aCopyObjectRequest
- Returns:
- A
Copy
that can be used to track the ongoing transfer - See Also:
copy(Consumer)
,S3AsyncClient.copyObject(CopyObjectRequest)
-
copy
default Copy copy(Consumer<CopyRequest.Builder> copyRequestBuilder)
This is a convenience method that creates an instance of theCopyRequest
builder, avoiding the need to create one manually viaCopyRequest.builder()
.- See Also:
copy(CopyRequest)
-
create
static S3TransferManager create()
Create anS3TransferManager
using the default values.The type of
S3AsyncClient
used depends on if AWS Common Runtime (CRT) librarysoftware.amazon.awssdk.crt:aws-crt
is in the classpath. If AWS CRT is available, an AWS CRT-based S3 client will be created via (S3AsyncClient.crtCreate()
). Otherwise, a standard S3 client(S3AsyncClient.create()
) will be created. Note that only AWS CRT-based S3 client supports parallel transfer, i.e, leveraging multipart upload/download for now, so it's recommended to add AWS CRT as a dependency.
-
builder
static S3TransferManager.Builder builder()
Creates a default builder forS3TransferManager
.
-
-