org.apache.hadoop.fs.s3a.commit.files

Class SinglePendingCommit

java.lang.Object

org.apache.hadoop.fs.s3a.commit.files.PersistentCommitData

org.apache.hadoop.fs.s3a.commit.files.SinglePendingCommit

All Implemented Interfaces:

Serializable, Iterable<String>, org.apache.hadoop.fs.statistics.IOStatisticsSource

@InterfaceAudience.Private
 @InterfaceStability.Unstable
public class SinglePendingCommit
extends PersistentCommitData
implements Iterable<String>, org.apache.hadoop.fs.statistics.IOStatisticsSource

This is the serialization format for uploads yet to be committed.

It's marked as Serializable so that it can be passed in RPC calls; for this to work it relies on the fact that java.io ArrayList and LinkedList are serializable. If any other list type is used for etags, it must also be serialized. Jackson expects lists, and it is used to persist to disk.

The statistics published through the IOStatisticsSource interface are the static ones marshalled with the commit data; they may be empty.

See Also:

Serialized Form

Field Summary

Fields inherited from class org.apache.hadoop.fs.s3a.commit.files.PersistentCommitData

VERSION

Constructor Summary

Constructors

Constructor and Description
SinglePendingCommit()

Method Summary

Modifier and Type	Method and Description
void	bindCommitData(List<com.amazonaws.services.s3.model.PartETag> parts) Set the commit data.
org.apache.hadoop.fs.Path	destinationPath() Build the destination path of the object.
String	getBucket()
long	getCreated() When was the upload created?
String	getDate() Timestamp as date; no expectation of parseability.
String	getDestinationKey()
List<String>	getEtags()
Map<String,String>	getExtraData() Any custom extra data committer subclasses may choose to add.
String	getFilename() This is the filename of the pending file itself.
org.apache.hadoop.fs.statistics.IOStatisticsSnapshot	getIOStatistics()
String	getJobId()
long	getLength() Destination file size.
int	getPartCount() Get the number of etags.
long	getSaved() When was the upload saved?
String	getTaskId()
String	getText() Arbitrary notes.
String	getUploadId()
String	getUri()
int	getVersion()
Iterator<String>	iterator() Iterate over the etags.
static SinglePendingCommit	load(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path) Load an instance from a file, then validate it.
void	putExtraData(String key, String value) Set/Update an extra data entry.
void	save(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path, boolean overwrite) Save to a hadoop filesystem.
static org.apache.hadoop.util.JsonSerialization<SinglePendingCommit>	serializer() Get a JSON serializer for this class.
void	setBucket(String bucket)
void	setCreated(long created)
void	setDate(String date)
void	setDestinationKey(String destinationKey)
void	setEtags(List<String> etags)
void	setExtraData(Map<String,String> extraData)
void	setFilename(String filename)
void	setIOStatistics(org.apache.hadoop.fs.statistics.IOStatisticsSnapshot ioStatistics)
void	setJobId(String jobId)
void	setLength(long length)
void	setSaved(long saved)
void	setTaskId(String taskId)
void	setText(String text)
void	setUploadId(String uploadId)
void	setUri(String uri)
void	setVersion(int version)
byte[]	toBytes() Serialize to JSON and then to a byte array, after performing a preflight validation of the data to be saved.
String	toString()
void	touch(long millis) Set the various timestamp fields to the supplied value.
void	validate() Validate the data: those fields which must be non empty, must be set.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface java.lang.Iterable

forEach, spliterator

Constructor Detail

SinglePendingCommit

public SinglePendingCommit()

Method Detail

serializer

public static org.apache.hadoop.util.JsonSerialization<SinglePendingCommit> serializer()

Get a JSON serializer for this class.

Returns:

a serializer.

load

public static SinglePendingCommit load(org.apache.hadoop.fs.FileSystem fs,
                                       org.apache.hadoop.fs.Path path)
                                throws IOException

Load an instance from a file, then validate it.

Parameters:

fs - filesystem

path - path

Returns:

the loaded instance

Throws:

IOException - IO failure

ValidationFailure - if the data is invalid

touch

public void touch(long millis)

Set the various timestamp fields to the supplied value.

Parameters:

millis - time in milliseconds

bindCommitData

public void bindCommitData(List<com.amazonaws.services.s3.model.PartETag> parts)
                    throws ValidationFailure

Set the commit data.

Parameters:

parts - ordered list of etags.

Throws:

ValidationFailure - if the data is invalid

validate

public void validate()
              throws ValidationFailure

Description copied from class: PersistentCommitData

Validate the data: those fields which must be non empty, must be set.

Specified by:

validate in class PersistentCommitData

Throws:

ValidationFailure - if the data is invalid

toString

public String toString()

Overrides:

toString in class Object

toBytes

public byte[] toBytes()
               throws IOException

Description copied from class: PersistentCommitData

Serialize to JSON and then to a byte array, after performing a preflight validation of the data to be saved.

Specified by:

toBytes in class PersistentCommitData

Returns:

the data in a persistable form.

Throws:

IOException - serialization problem or validation failure.

save

public void save(org.apache.hadoop.fs.FileSystem fs,
                 org.apache.hadoop.fs.Path path,
                 boolean overwrite)
          throws IOException

Description copied from class: PersistentCommitData

Save to a hadoop filesystem.

Specified by:

save in class PersistentCommitData

Parameters:

fs - filesystem

path - path

overwrite - should any existing file be overwritten

Throws:

IOException - IO exception

destinationPath

public org.apache.hadoop.fs.Path destinationPath()

Build the destination path of the object.

Returns:

the path

Throws:

IllegalStateException - if the URI is invalid

getPartCount

public int getPartCount()

Get the number of etags.

Returns:

the size of the etag list.

iterator

public Iterator<String> iterator()

Iterate over the etags.

Specified by:

iterator in interface Iterable<String>

Returns:

an iterator.

getVersion

public int getVersion()

Returns:

version marker.

setVersion

public void setVersion(int version)

getFilename

public String getFilename()

This is the filename of the pending file itself. Used during processing; it's persistent value, if any, is ignored.

Returns:

filename

setFilename

public void setFilename(String filename)

getUri

public String getUri()

Returns:

path URI of the destination.

setUri

public void setUri(String uri)

getUploadId

public String getUploadId()

Returns:

ID of the upload.

setUploadId

public void setUploadId(String uploadId)

getBucket

public String getBucket()

Returns:

destination bucket.

setBucket

public void setBucket(String bucket)

getDestinationKey

public String getDestinationKey()

Returns:

destination key in the bucket.

setDestinationKey

public void setDestinationKey(String destinationKey)

getCreated

public long getCreated()

When was the upload created?

Returns:

timestamp

setCreated

public void setCreated(long created)

getSaved

public long getSaved()

When was the upload saved?

Returns:

timestamp

setSaved

public void setSaved(long saved)

getDate

public String getDate()

Timestamp as date; no expectation of parseability.

Returns:

date string

setDate

public void setDate(String date)

getJobId

public String getJobId()

Returns:

Job ID, if known.

setJobId

public void setJobId(String jobId)

getTaskId

public String getTaskId()

Returns:

Task ID, if known.

setTaskId

public void setTaskId(String taskId)

getText

public String getText()

Arbitrary notes.

Returns:

any notes

setText

public void setText(String text)

getEtags

public List<String> getEtags()

Returns:

ordered list of etags.

setEtags

public void setEtags(List<String> etags)

getExtraData

public Map<String,String> getExtraData()

Any custom extra data committer subclasses may choose to add.

Returns:

custom data

setExtraData

public void setExtraData(Map<String,String> extraData)

putExtraData

public void putExtraData(String key,
                         String value)

Set/Update an extra data entry.

Parameters:

key - key

value - value

getLength

public long getLength()

Destination file size.

Returns:

size of destination object

setLength

public void setLength(long length)

getIOStatistics

public org.apache.hadoop.fs.statistics.IOStatisticsSnapshot getIOStatistics()

Specified by:

getIOStatistics in interface org.apache.hadoop.fs.statistics.IOStatisticsSource

setIOStatistics

public void setIOStatistics(org.apache.hadoop.fs.statistics.IOStatisticsSnapshot ioStatistics)