com.netflix.priam.backup

Class AbstractFileSystem

java.lang.Object

com.netflix.priam.backup.AbstractFileSystem

All Implemented Interfaces:

IBackupFileSystem, EventGenerator<BackupEvent>

Direct Known Subclasses:

GoogleEncryptedFileSystem, S3FileSystemBase

public abstract class AbstractFileSystem
extends java.lang.Object
implements IBackupFileSystem, EventGenerator<BackupEvent>

This class is responsible for managing parallelism and orchestrating the upload and download, but the subclasses actually implement the details of uploading a file.

Created by aagrawal on 8/30/18.

Field Summary

Fields

Modifier and Type	Field and Description
protected BackupMetrics	backupMetrics
protected com.google.inject.Provider<AbstractBackupPath>	pathProvider

Constructor Summary

Constructors

Constructor and Description
AbstractFileSystem(IConfiguration configuration, BackupMetrics backupMetrics, BackupNotificationMgr backupNotificationMgr, com.google.inject.Provider<AbstractBackupPath> pathProvider)

Method Summary

Modifier and Type	Method and Description
void	addObserver(EventObserver<BackupEvent> observer) Subscribes observer to receive generated events.
java.util.concurrent.Future<java.nio.file.Path>	asyncDownloadFile(AbstractBackupPath path, int retry) Download the file denoted by remotePath in an async fashion to the local file system denoted by local path.
com.google.common.util.concurrent.ListenableFuture<AbstractBackupPath>	asyncUploadAndDelete(AbstractBackupPath path, int retry) Upload the local file denoted by localPath in async fashion to the remote file system at location denoted by remotePath.
boolean	checkObjectExists(java.nio.file.Path remotePath) Checks if the file denoted by remotePath exists on the remote file system.
void	clearCache() Clear the cache for the backup file system, if any.
protected abstract void	deleteFiles(java.util.List<java.nio.file.Path> remotePaths)
void	deleteRemoteFiles(java.util.List<java.nio.file.Path> remotePaths) Delete list of remote files from the remote file system.
protected abstract boolean	doesRemoteFileExist(java.nio.file.Path remotePath)
void	downloadFile(AbstractBackupPath path, java.lang.String suffix, int retry) Download the file denoted by remotePath to the local file system denoted by local path.
protected abstract void	downloadFileImpl(AbstractBackupPath path, java.lang.String suffix)
int	getDownloadTasksQueued() Get the number of tasks en-queue in the filesystem for download.
java.nio.file.Path	getPrefix() Get the prefix path for the backup file system.
java.lang.String	getShard() Get the shard where this object should be stored.
int	getUploadTasksQueued() Get the number of tasks en-queue in the filesystem for upload.
java.util.Iterator<AbstractBackupPath>	list(java.lang.String path, java.util.Date start, java.util.Date till) List all files in the backup location for the specified time range.
java.util.Iterator<AbstractBackupPath>	listPrefixes(java.util.Date date) Get a list of prefixes for the cluster available in backup for the specified date
void	notifyEventFailure(BackupEvent event) Notify all the observers of an event failure who subscribed to receive events.
void	notifyEventStart(BackupEvent event) Notify all the observers of an event start who subscribed to receive events.
void	notifyEventStop(BackupEvent event) Notify all the observers of an event stop who subscribed to receive events.
void	notifyEventSuccess(BackupEvent event) Notify all the observers of an event success who subscribed to receive events.
void	removeObserver(EventObserver<BackupEvent> observer) Removes observer from receiving any further events from this generator.
void	uploadAndDelete(AbstractBackupPath path, int retry) Upload the local file to its remote counterpart.
protected abstract long	uploadFileImpl(AbstractBackupPath path)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface com.netflix.priam.backup.IBackupFileSystem

cleanup, getFileSize, listFileSystem, shutdown

Field Detail

pathProvider

protected final com.google.inject.Provider<AbstractBackupPath> pathProvider

backupMetrics

protected final BackupMetrics backupMetrics

Constructor Detail

AbstractFileSystem

@Inject
public AbstractFileSystem(IConfiguration configuration,
                                  BackupMetrics backupMetrics,
                                  BackupNotificationMgr backupNotificationMgr,
                                  com.google.inject.Provider<AbstractBackupPath> pathProvider)

Method Detail

asyncDownloadFile

public java.util.concurrent.Future<java.nio.file.Path> asyncDownloadFile(AbstractBackupPath path,
                                                                         int retry)
                                                                  throws java.util.concurrent.RejectedExecutionException

Description copied from interface: IBackupFileSystem

Download the file denoted by remotePath in an async fashion to the local file system denoted by local path.

Specified by:

asyncDownloadFile in interface IBackupFileSystem

Parameters:

path - Backup path representing a local and remote file pair

retry - No. of times to retry to download a file from remote file system. If <1, it will try to download file exactly once.

Returns:

The future of the async job to monitor the progress of the job.

Throws:

java.util.concurrent.RejectedExecutionException - if the queue is full and TIMEOUT is reached while trying to add the work to the queue.

downloadFile

public void downloadFile(AbstractBackupPath path,
                         java.lang.String suffix,
                         int retry)
                  throws BackupRestoreException

Description copied from interface: IBackupFileSystem

Download the file denoted by remotePath to the local file system denoted by local path.

Specified by:

downloadFile in interface IBackupFileSystem

Parameters:

path - Backup path representing a local and remote file pair

retry - No. of times to retry to download a file from remote file system. If <1, it will try to download file exactly once.

Throws:

BackupRestoreException - if file is not available, downloadable or any other error from remote file system.

downloadFileImpl

protected abstract void downloadFileImpl(AbstractBackupPath path,
                                         java.lang.String suffix)
                                  throws BackupRestoreException

Throws:

BackupRestoreException

asyncUploadAndDelete

public com.google.common.util.concurrent.ListenableFuture<AbstractBackupPath> asyncUploadAndDelete(AbstractBackupPath path,
                                                                                                   int retry)
                                                                                            throws java.util.concurrent.RejectedExecutionException

Description copied from interface: IBackupFileSystem

Upload the local file denoted by localPath in async fashion to the remote file system at location denoted by remotePath.

Specified by:

asyncUploadAndDelete in interface IBackupFileSystem

Parameters:

path - AbstractBackupPath to be used to send backup notifications only.

retry - No of times to retry to upload a file. If <1, it will try to upload file exactly once.

Returns:

The future of the async job to monitor the progress of the job. This will be null if file was de-duped for upload.

Throws:

java.util.concurrent.RejectedExecutionException - if the queue is full and TIMEOUT is reached while trying to add the work to the queue.

uploadAndDelete

public void uploadAndDelete(AbstractBackupPath path,
                            int retry)
                     throws BackupRestoreException

Description copied from interface: IBackupFileSystem

Upload the local file to its remote counterpart. Both locations are embedded within the path parameter. De-duping of the file to upload will always be done by comparing the files-in-progress to be uploaded. This may result in this particular request to not to be executed e.g. if any other thread has given the same file to upload and that file is in internal queue. Note that de-duping is best effort and is not always guaranteed as we try to avoid lock on read/write of the files-in-progress. Once uploaded, files are deleted.

Specified by:

uploadAndDelete in interface IBackupFileSystem

Parameters:

path - Backup path representing a local and remote file pair

retry - No of times to retry to upload a file. If <1, it will try to upload file exactly once.

Throws:

BackupRestoreException - in case of failure to upload for any reason including file not readable or remote file system errors.

checkObjectExists

public boolean checkObjectExists(java.nio.file.Path remotePath)

Description copied from interface: IBackupFileSystem

Checks if the file denoted by remotePath exists on the remote file system. It does not need check if object was completely uploaded to remote file system.

Specified by:

checkObjectExists in interface IBackupFileSystem

Parameters:

remotePath - location on the remote file system.

Returns:

boolean value indicating presence of the file on remote file system.

deleteRemoteFiles

public void deleteRemoteFiles(java.util.List<java.nio.file.Path> remotePaths)
                       throws BackupRestoreException

Description copied from interface: IBackupFileSystem

Delete list of remote files from the remote file system. It should throw exception if there is anything wrong in processing the request. If the remotePath passed do not exist, then it should just keep quiet.

Specified by:

deleteRemoteFiles in interface IBackupFileSystem

Parameters:

remotePaths - list of files on remote file system to be deleted. This path may or may not exist.

Throws:

BackupRestoreException - in case of remote file system not able to process the request or unable to reach.

deleteFiles

protected abstract void deleteFiles(java.util.List<java.nio.file.Path> remotePaths)
                             throws BackupRestoreException

Throws:

BackupRestoreException

doesRemoteFileExist

protected abstract boolean doesRemoteFileExist(java.nio.file.Path remotePath)

uploadFileImpl

protected abstract long uploadFileImpl(AbstractBackupPath path)
                                throws BackupRestoreException

Throws:

BackupRestoreException

getShard

public java.lang.String getShard()

Description copied from interface: IBackupFileSystem

Get the shard where this object should be stored. For local file system this should be empty or null. For S3, it would be the location of the bucket.

Specified by:

getShard in interface IBackupFileSystem

Returns:

the location of the shard.

getPrefix

public java.nio.file.Path getPrefix()

Description copied from interface: IBackupFileSystem

Get the prefix path for the backup file system. This will be either the location of the remote file system for backup or the location from where we should restore.

Specified by:

getPrefix in interface IBackupFileSystem

Returns:

prefix path to the backup file system.

listPrefixes

public java.util.Iterator<AbstractBackupPath> listPrefixes(java.util.Date date)

Description copied from interface: IBackupFileSystem

Get a list of prefixes for the cluster available in backup for the specified date

Specified by:

listPrefixes in interface IBackupFileSystem

list

public java.util.Iterator<AbstractBackupPath> list(java.lang.String path,
                                                   java.util.Date start,
                                                   java.util.Date till)

Description copied from interface: IBackupFileSystem

List all files in the backup location for the specified time range.

Specified by:

list in interface IBackupFileSystem

Parameters:

path - This is used as the `prefix` for listing files in the filesystem. All the files that start with this prefix will be returned.

start - Start date of the file upload.

till - End date of the file upload.

Returns:

Iterator of the AbstractBackupPath matching the criteria.

addObserver

public final void addObserver(EventObserver<BackupEvent> observer)

Description copied from interface: EventGenerator

Subscribes observer to receive generated events.

Specified by:

addObserver in interface EventGenerator<BackupEvent>

Parameters:

observer - EventObserver interested in receiving updates from this event generator. May not be null.

removeObserver

public void removeObserver(EventObserver<BackupEvent> observer)

Description copied from interface: EventGenerator

Removes observer from receiving any further events from this generator.

Specified by:

removeObserver in interface EventGenerator<BackupEvent>

Parameters:

observer - EventObserver that is to stop receiving updates from this event generator. May not be null.

notifyEventStart

public void notifyEventStart(BackupEvent event)

Description copied from interface: EventGenerator

Notify all the observers of an event start who subscribed to receive events.

Specified by:

notifyEventStart in interface EventGenerator<BackupEvent>

Parameters:

event - Generated event from the generator

notifyEventSuccess

public void notifyEventSuccess(BackupEvent event)

Description copied from interface: EventGenerator

Notify all the observers of an event success who subscribed to receive events.

Specified by:

notifyEventSuccess in interface EventGenerator<BackupEvent>

Parameters:

event - Event which was successful from the generator.

notifyEventFailure

public void notifyEventFailure(BackupEvent event)

Description copied from interface: EventGenerator

Notify all the observers of an event failure who subscribed to receive events.

Specified by:

notifyEventFailure in interface EventGenerator<BackupEvent>

Parameters:

event - Event which was failure from the generator.

notifyEventStop

public void notifyEventStop(BackupEvent event)

Description copied from interface: EventGenerator

Notify all the observers of an event stop who subscribed to receive events.

Specified by:

notifyEventStop in interface EventGenerator<BackupEvent>

Parameters:

event - Event which was stopped from the generator.

getUploadTasksQueued

public int getUploadTasksQueued()

Description copied from interface: IBackupFileSystem

Get the number of tasks en-queue in the filesystem for upload.

Specified by:

getUploadTasksQueued in interface IBackupFileSystem

Returns:

the total no. of tasks to be executed.

getDownloadTasksQueued

public int getDownloadTasksQueued()

Description copied from interface: IBackupFileSystem

Get the number of tasks en-queue in the filesystem for download.

Specified by:

getDownloadTasksQueued in interface IBackupFileSystem

Returns:

the total no. of tasks to be executed.

clearCache

public void clearCache()

Description copied from interface: IBackupFileSystem

Clear the cache for the backup file system, if any.

Specified by:

clearCache in interface IBackupFileSystem