org.apache.hadoop.fs.s3a.impl

Class CopyFromLocalOperation

java.lang.Object

org.apache.hadoop.fs.s3a.impl.AbstractStoreOperation

org.apache.hadoop.fs.s3a.impl.ExecutingStoreOperation<Void>

org.apache.hadoop.fs.s3a.impl.CopyFromLocalOperation

All Implemented Interfaces:

org.apache.hadoop.util.functional.CallableRaisingIOE<Void>

public class CopyFromLocalOperation
extends ExecutingStoreOperation<Void>

Implementation of CopyFromLocalOperation.

This operation copies a file or directory (recursively) from a local FS to an object store. Initially, this operation has been developed for S3 (s3a) interaction, however, there's minimal work needed for it to work with other stores.

How the uploading of files works:

• all source files and directories are scanned through;
• the LARGEST_N_FILES start uploading;
• the remaining files are shuffled and uploaded;
• any remaining empty directory is uploaded too to preserve local tree structure.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static interface	CopyFromLocalOperation.CopyFromLocalOperationCallbacks Define the contract for CopyFromLocalOperation to interact with any external resources.

Constructor Summary

Constructors

Constructor and Description
CopyFromLocalOperation(StoreContext storeContext, org.apache.hadoop.fs.Path source, org.apache.hadoop.fs.Path destination, boolean deleteSource, boolean overwrite, CopyFromLocalOperation.CopyFromLocalOperationCallbacks callbacks)

Method Summary

Modifier and Type	Method and Description
Void	execute() Executes the CopyFromLocalOperation.

Methods inherited from class org.apache.hadoop.fs.s3a.impl.ExecutingStoreOperation

apply, executeOnlyOnce

Methods inherited from class org.apache.hadoop.fs.s3a.impl.AbstractStoreOperation

activateAuditSpan, getAuditSpan, getStoreContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

CopyFromLocalOperation

public CopyFromLocalOperation(StoreContext storeContext,
                              org.apache.hadoop.fs.Path source,
                              org.apache.hadoop.fs.Path destination,
                              boolean deleteSource,
                              boolean overwrite,
                              CopyFromLocalOperation.CopyFromLocalOperationCallbacks callbacks)

Method Detail

execute

@Retries.RetryTranslated
public Void execute()
                                      throws IOException,
                                             org.apache.hadoop.fs.PathExistsException

Executes the CopyFromLocalOperation.

Specified by:

execute in class ExecutingStoreOperation<Void>

Returns:

the result.

Throws:

IOException - - if there are any failures with upload or deletion of files. Check CopyFromLocalOperation.CopyFromLocalOperationCallbacks for specifics.

org.apache.hadoop.fs.PathExistsException - - if the path exists and no overwrite flag is set OR if the source is file and destination is a directory