tachyon.master

Class MasterInfo

java.lang.Object

tachyon.master.ImageWriter

tachyon.master.MasterInfo

public class MasterInfo
extends ImageWriter

A global view of filesystem in master.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
class	MasterInfo.MasterInfoHeartbeatExecutor Master info periodical status check.
class	MasterInfo.RecomputationScheduler

Field Summary

Fields

Modifier and Type	Field and Description
static String	COL

Constructor Summary

Constructors

Constructor and Description
MasterInfo(InetSocketAddress address, Journal journal, ExecutorService mExecutorService)

Method Summary

Methods

Modifier and Type	Method and Description
ClientRawTableInfo	_getClientRawTableInfo(TachyonURI path, Inode inode) Get the raw table info associated with the given id.
boolean	_rename(int fileId, TachyonURI dstPath, long opTimeMs) Rename a file to the given path, inner method.
boolean	addCheckpoint(long workerId, int fileId, long length, TachyonURI checkpointPath) Add a checkpoint to a file.
int	cacheBlock(long workerId, long workerUsedBytes, long storageDirId, long blockId, long length) A worker cache a block in its memory.
void	completeFile(int fileId) Completes the checkpointing of a file.
int	createDependency(List<TachyonURI> parents, List<TachyonURI> children, String commandPrefix, List<ByteBuffer> data, String comment, String framework, String frameworkVersion, DependencyType dependencyType)
int	createFile(boolean recursive, TachyonURI path, boolean directory, long blockSizeByte) Create a file.
int	createFile(TachyonURI path, long blockSizeByte)
int	createFile(TachyonURI path, long blockSizeByte, boolean recursive)
long	createNewBlock(int fileId) Creates a new block for the given file.
int	createRawTable(TachyonURI path, int columns, ByteBuffer metadata) Creates a raw table.
boolean	delete(int fileId, boolean recursive) Delete a file based on the file's ID.
boolean	delete(TachyonURI path, boolean recursive) Delete files based on the path.
boolean	freepath(TachyonURI path, boolean recursive) Frees files based on the path
long	getBlockIdBasedOnOffset(int fileId, long offset)
List<BlockInfo>	getBlockList(TachyonURI path) Get the list of blocks of an InodeFile determined by path.
long	getCapacityBytes() Get the capacity of the whole system.
ClientBlockInfo	getClientBlockInfo(long blockId) Get the block info associated with the given id.
ClientDependencyInfo	getClientDependencyInfo(int dependencyId) Get the dependency info associated with the given id.
ClientFileInfo	getClientFileInfo(int fid) Get the file info associated with the given id.
ClientFileInfo	getClientFileInfo(TachyonURI path) Get the file info for the file at the given path
ClientRawTableInfo	getClientRawTableInfo(int id) Get the raw table info associated with the given id.
ClientRawTableInfo	getClientRawTableInfo(TachyonURI path) Get the raw table info for the table at the given path
List<ClientBlockInfo>	getFileBlocks(int fileId) Get the block infos of a file with the given id.
List<ClientBlockInfo>	getFileBlocks(TachyonURI path) Get the block infos of a file with the given path.
int	getFileId(TachyonURI path) Get the file id of the file.
List<ClientFileInfo>	getFilesInfo(TachyonURI path) If the path is a directory, return all the direct entries in it.
List<TachyonURI>	getInMemoryFiles() Get absolute paths of all in memory files.
Journal	getJournal() Get Journal instance for MasterInfo for Unit test only
List<ClientWorkerInfo>	getLostWorkersInfo() Get info about the lost workers
InetSocketAddress	getMasterAddress() Get the master address.
long	getNewUserId() Get a new user id
int	getNumberOfFiles(TachyonURI path) Get the number of files at a given path.
TachyonURI	getPath(int fileId) Get the path of a file with the given id
List<Integer>	getPinIdList() Get a list of the pin id's.
List<Integer>	getPriorityDependencyList() Creates a list of high priority dependencies, which don't yet have checkpoints.
int	getRawTableId(TachyonURI path) Get the id of the table at the given path.
long	getStarttimeMs() Get the master start time in milliseconds.
long	getUnderFsCapacityBytes() Get the capacity of the under file system.
long	getUnderFsFreeBytes() Get the amount of free space in the under file system.
long	getUnderFsUsedBytes() Get the amount of space used in the under file system.
long	getUsedBytes() Get the amount of space used by the workers.
List<String>	getWhiteList() Get the white list.
NetAddress	getWorker(boolean random, String host) Get the address of a worker.
int	getWorkerCount() Get the number of workers.
List<ClientWorkerInfo>	getWorkersInfo() Get info about all the workers.
void	init()
List<Integer>	listFiles(TachyonURI path, boolean recursive) Get the id of the file at the given path.
void	loadImage(com.fasterxml.jackson.core.JsonParser parser, TachyonURI path) Load the image from parser, which is created based on the path.
List<TachyonURI>	ls(TachyonURI path, boolean recursive) Get the names of the sub-directories at the given path.
boolean	mkdirs(TachyonURI path, boolean recursive) Create a directory at the given path.
long	registerWorker(NetAddress workerNetAddress, long totalBytes, long usedBytes, Map<Long,List<Long>> currentBlockIds) Register a worker at the given address, setting it up and associating it with a given list of blocks.
boolean	rename(int fileId, TachyonURI dstPath) Rename a file to the given path.
boolean	rename(TachyonURI srcPath, TachyonURI dstPath) Rename a file to the given path.
void	reportLostFile(int fileId) Logs a lost file and sets it to be recovered.
void	requestFilesInDependency(int depId) Request that the files for the given dependency be recomputed.
void	setPinned(int fileId, boolean pinned) Sets the isPinned flag on the given inode and all of its children.
void	stop() Stops the heartbeat thread.
void	updateRawTableMetadata(int tableId, ByteBuffer metadata) Update the metadata of a table.
Command	workerHeartbeat(long workerId, long usedBytes, List<Long> removedBlockIds, Map<Long,List<Long>> addedBlockIds) The heartbeat of the worker.
void	writeImage(com.fasterxml.jackson.databind.ObjectWriter objWriter, DataOutputStream dos) Create an image of the dependencies and filesystem tree.

Methods inherited from class tachyon.master.ImageWriter

writeElement

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

COL

public static final String COL

See Also:

Constant Field Values

Constructor Detail

MasterInfo

public MasterInfo(InetSocketAddress address,
          Journal journal,
          ExecutorService mExecutorService)
           throws IOException

Throws:

IOException

Method Detail

_getClientRawTableInfo

public ClientRawTableInfo _getClientRawTableInfo(TachyonURI path,
                                        Inode inode)
                                          throws TableDoesNotExistException

Get the raw table info associated with the given id.

Parameters:

path - The path of the table

inode - The inode at the path

Returns:

the table info

Throws:

TableDoesNotExistException

_rename

public boolean _rename(int fileId,
              TachyonURI dstPath,
              long opTimeMs)
                throws FileDoesNotExistException,
                       InvalidPathException

Rename a file to the given path, inner method.

Parameters:

fileId - The id of the file to rename

dstPath - The new path of the file

opTimeMs - The time of the rename operation, in milliseconds

Returns:

true if the rename succeeded, false otherwise

Throws:

FileDoesNotExistException - If the id doesn't point to an inode

InvalidPathException - if the source path is a prefix of the destination

addCheckpoint

public boolean addCheckpoint(long workerId,
                    int fileId,
                    long length,
                    TachyonURI checkpointPath)
                      throws FileNotFoundException,
                             SuspectedFileSizeException,
                             BlockInfoException

Add a checkpoint to a file.

Parameters:

workerId - The worker which submitted the request. -1 if the request is not from a worker.

fileId - The file to add the checkpoint.

length - The length of the checkpoint.

checkpointPath - The path of the checkpoint.

Returns:

true if the checkpoint is added successfully, false if not.

Throws:

FileNotFoundException

SuspectedFileSizeException

BlockInfoException

cacheBlock

public int cacheBlock(long workerId,
             long workerUsedBytes,
             long storageDirId,
             long blockId,
             long length)
               throws FileDoesNotExistException,
                      SuspectedFileSizeException,
                      BlockInfoException

A worker cache a block in its memory.

Parameters:

workerId -

workerUsedBytes -

blockId -

length -

Returns:

the dependency id of the file if it has not been checkpointed. -1 means the file either does not have dependency or has already been checkpointed.

Throws:

FileDoesNotExistException

SuspectedFileSizeException

BlockInfoException

completeFile

public void completeFile(int fileId)
                  throws FileDoesNotExistException

Completes the checkpointing of a file.

Parameters:

fileId - The id of the file

Throws:

FileDoesNotExistException

createDependency

public int createDependency(List<TachyonURI> parents,
                   List<TachyonURI> children,
                   String commandPrefix,
                   List<ByteBuffer> data,
                   String comment,
                   String framework,
                   String frameworkVersion,
                   DependencyType dependencyType)
                     throws InvalidPathException,
                            FileDoesNotExistException

Throws:

InvalidPathException

FileDoesNotExistException

createFile

public int createFile(boolean recursive,
             TachyonURI path,
             boolean directory,
             long blockSizeByte)
               throws FileAlreadyExistException,
                      InvalidPathException,
                      BlockInfoException,
                      TachyonException

Create a file. // TODO Make this API better.

Throws:

FileAlreadyExistException

InvalidPathException

BlockInfoException

TachyonException

createFile

public int createFile(TachyonURI path,
             long blockSizeByte)
               throws FileAlreadyExistException,
                      InvalidPathException,
                      BlockInfoException,
                      TachyonException

Throws:

FileAlreadyExistException

InvalidPathException

BlockInfoException

TachyonException

createFile

public int createFile(TachyonURI path,
             long blockSizeByte,
             boolean recursive)
               throws FileAlreadyExistException,
                      InvalidPathException,
                      BlockInfoException,
                      TachyonException

Throws:

FileAlreadyExistException

InvalidPathException

BlockInfoException

TachyonException

createNewBlock

public long createNewBlock(int fileId)
                    throws FileDoesNotExistException

Creates a new block for the given file.

Parameters:

fileId - The id of the file

Returns:

the block id.

Throws:

FileDoesNotExistException

createRawTable

public int createRawTable(TachyonURI path,
                 int columns,
                 ByteBuffer metadata)
                   throws FileAlreadyExistException,
                          InvalidPathException,
                          TableColumnException,
                          TachyonException

Creates a raw table.

Parameters:

path - The path to place the table at

columns - The number of columns in the table

metadata - Additional metadata about the table

Returns:

the file id of the table

Throws:

FileAlreadyExistException

InvalidPathException

TableColumnException

TachyonException

delete

public boolean delete(int fileId,
             boolean recursive)
               throws TachyonException

Delete a file based on the file's ID.

Parameters:

fileId - the file to be deleted.

recursive - whether delete the file recursively or not.

Returns:

succeed or not

Throws:

TachyonException

delete

public boolean delete(TachyonURI path,
             boolean recursive)
               throws TachyonException

Delete files based on the path.

Parameters:

path - The file to be deleted.

recursive - whether delete the file recursively or not.

Returns:

succeed or not

Throws:

TachyonException

getBlockIdBasedOnOffset

public long getBlockIdBasedOnOffset(int fileId,
                           long offset)
                             throws FileDoesNotExistException

Throws:

FileDoesNotExistException

getBlockList

public List<BlockInfo> getBlockList(TachyonURI path)
                             throws InvalidPathException,
                                    FileDoesNotExistException

Get the list of blocks of an InodeFile determined by path.

Parameters:

path - The file.

Returns:

The list of the blocks of the file.

Throws:

InvalidPathException

FileDoesNotExistException

getCapacityBytes

public long getCapacityBytes()

Get the capacity of the whole system.

Returns:

the system's capacity in bytes.

getClientBlockInfo

public ClientBlockInfo getClientBlockInfo(long blockId)
                                   throws FileDoesNotExistException,
                                          BlockInfoException

Get the block info associated with the given id.

Parameters:

blockId - The id of the block return

Returns:

the block info

Throws:

FileDoesNotExistException

BlockInfoException

getClientDependencyInfo

public ClientDependencyInfo getClientDependencyInfo(int dependencyId)
                                             throws DependencyDoesNotExistException

Get the dependency info associated with the given id.

Parameters:

dependencyId - The id of the dependency

Returns:

the dependency info

Throws:

DependencyDoesNotExistException

getClientFileInfo

public ClientFileInfo getClientFileInfo(int fid)

Get the file info associated with the given id.

Parameters:

fid - The id of the file

Returns:

the file info

getClientFileInfo

public ClientFileInfo getClientFileInfo(TachyonURI path)
                                 throws InvalidPathException

Get the file info for the file at the given path

Parameters:

path - The path of the file

Returns:

the file info

Throws:

InvalidPathException

getClientRawTableInfo

public ClientRawTableInfo getClientRawTableInfo(int id)
                                         throws TableDoesNotExistException

Get the raw table info associated with the given id.

Parameters:

id - The id of the table

Returns:

the table info

Throws:

TableDoesNotExistException

getClientRawTableInfo

public ClientRawTableInfo getClientRawTableInfo(TachyonURI path)
                                         throws TableDoesNotExistException,
                                                InvalidPathException

Get the raw table info for the table at the given path

Parameters:

path - The path of the table

Returns:

the table info

Throws:

TableDoesNotExistException

InvalidPathException

getFileId

public int getFileId(TachyonURI path)
              throws InvalidPathException

Get the file id of the file.

Parameters:

path - The path of the file

Returns:

The file id of the file. -1 if the file does not exist.

Throws:

InvalidPathException

getFileBlocks

public List<ClientBlockInfo> getFileBlocks(int fileId)
                                    throws FileDoesNotExistException

Get the block infos of a file with the given id. Throws an exception if the id names a directory.

Parameters:

fileId - The id of the file to look up

Returns:

the block infos of the file

Throws:

FileDoesNotExistException

getFileBlocks

public List<ClientBlockInfo> getFileBlocks(TachyonURI path)
                                    throws FileDoesNotExistException,
                                           InvalidPathException

Get the block infos of a file with the given path. Throws an exception if the path names a directory.

Parameters:

path - The path of the file to look up

Returns:

the block infos of the file

Throws:

FileDoesNotExistException

InvalidPathException

getFilesInfo

public List<ClientFileInfo> getFilesInfo(TachyonURI path)
                                  throws FileDoesNotExistException,
                                         InvalidPathException

If the path is a directory, return all the direct entries in it. If the path is a file, return its ClientFileInfo.

Parameters:

path - the target directory/file path

Returns:

A list of ClientFileInfo

Throws:

FileDoesNotExistException

InvalidPathException

getInMemoryFiles

public List<TachyonURI> getInMemoryFiles()

Get absolute paths of all in memory files.

Returns:

absolute paths of all in memory files.

getJournal

public Journal getJournal()

Get Journal instance for MasterInfo for Unit test only

Returns:

Journal instance

getMasterAddress

public InetSocketAddress getMasterAddress()

Get the master address.

Returns:

the master address

getNewUserId

public long getNewUserId()

Get a new user id

Returns:

a new user id

getNumberOfFiles

public int getNumberOfFiles(TachyonURI path)
                     throws InvalidPathException,
                            FileDoesNotExistException

Get the number of files at a given path.

Parameters:

path - The path to look at

Returns:

The number of files at the path. Returns 1 if the path specifies a file. If it's a directory, returns the number of items in the directory.

Throws:

InvalidPathException

FileDoesNotExistException

getPath

public TachyonURI getPath(int fileId)
                   throws FileDoesNotExistException

Get the path of a file with the given id

Parameters:

fileId - The id of the file to look up

Returns:

the path of the file

Throws:

FileDoesNotExistException - raise if the file does not exist.

getPinIdList

public List<Integer> getPinIdList()

Get a list of the pin id's.

Returns:

a list of pin id's

getPriorityDependencyList

public List<Integer> getPriorityDependencyList()

Creates a list of high priority dependencies, which don't yet have checkpoints.

Returns:

the list of dependency ids

getRawTableId

public int getRawTableId(TachyonURI path)
                  throws InvalidPathException,
                         TableDoesNotExistException

Get the id of the table at the given path.

Parameters:

path - The path of the table

Returns:

the id of the table

Throws:

InvalidPathException

TableDoesNotExistException

getStarttimeMs

public long getStarttimeMs()

Get the master start time in milliseconds.

Returns:

the master start time in milliseconds

getUnderFsCapacityBytes

public long getUnderFsCapacityBytes()
                             throws IOException

Get the capacity of the under file system.

Returns:

the capacity in bytes

Throws:

IOException

getUnderFsFreeBytes

public long getUnderFsFreeBytes()
                         throws IOException

Get the amount of free space in the under file system.

Returns:

the free space in bytes

Throws:

IOException

getUnderFsUsedBytes

public long getUnderFsUsedBytes()
                         throws IOException

Get the amount of space used in the under file system.

Returns:

the space used in bytes

Throws:

IOException

getUsedBytes

public long getUsedBytes()

Get the amount of space used by the workers.

Returns:

the amount of space used in bytes

getWhiteList

public List<String> getWhiteList()

Get the white list.

Returns:

the white list

getWorker

public NetAddress getWorker(boolean random,
                   String host)
                     throws UnknownHostException

Get the address of a worker.

Parameters:

random - If true, select a random worker

host - If random is false, select a worker on this host

Returns:

the address of the selected worker, or null if no address could be found

Throws:

UnknownHostException

getWorkerCount

public int getWorkerCount()

Get the number of workers.

Returns:

the number of workers

getWorkersInfo

public List<ClientWorkerInfo> getWorkersInfo()

Get info about all the workers.

Returns:

a list of worker infos

getLostWorkersInfo

public List<ClientWorkerInfo> getLostWorkersInfo()

Get info about the lost workers

Returns:

a list of worker info

init

public void init()
          throws IOException

Throws:

IOException

listFiles

public List<Integer> listFiles(TachyonURI path,
                      boolean recursive)
                        throws InvalidPathException,
                               FileDoesNotExistException

Get the id of the file at the given path. If recursive, it scans the subdirectories as well.

Parameters:

path - The path to start looking at

recursive - If true, recursively scan the subdirectories at the given path as well

Returns:

the list of the inode id's at the path

Throws:

InvalidPathException

FileDoesNotExistException

loadImage

public void loadImage(com.fasterxml.jackson.core.JsonParser parser,
             TachyonURI path)
               throws IOException

Load the image from parser, which is created based on the path. Assume this blocks the whole MasterInfo.

Parameters:

parser - the JsonParser to load the image

path - the file to load the image

Throws:

IOException

ls

public List<TachyonURI> ls(TachyonURI path,
                  boolean recursive)
                    throws InvalidPathException,
                           FileDoesNotExistException

Get the names of the sub-directories at the given path.

Parameters:

path - The path to look at

recursive - If true, recursively add the paths of the sub-directories

Returns:

the list of paths

Throws:

InvalidPathException

FileDoesNotExistException

mkdirs

public boolean mkdirs(TachyonURI path,
             boolean recursive)
               throws FileAlreadyExistException,
                      InvalidPathException,
                      TachyonException

Create a directory at the given path.

Parameters:

path - The path to create a directory at

Returns:

true if and only if the directory was created; false otherwise

Throws:

FileAlreadyExistException

InvalidPathException

TachyonException

registerWorker

public long registerWorker(NetAddress workerNetAddress,
                  long totalBytes,
                  long usedBytes,
                  Map<Long,List<Long>> currentBlockIds)
                    throws BlockInfoException

Register a worker at the given address, setting it up and associating it with a given list of blocks.

Parameters:

workerNetAddress - The address of the worker to register

totalBytes - The capacity of the worker in bytes

usedBytes - The number of bytes already used in the worker

currentBlockIds - Mapping from id of the StorageDir to id list of the blocks

Returns:

the new id of the registered worker

Throws:

BlockInfoException

rename

public boolean rename(int fileId,
             TachyonURI dstPath)
               throws FileDoesNotExistException,
                      InvalidPathException

Rename a file to the given path.

Parameters:

fileId - The id of the file to rename

dstPath - The new path of the file

Returns:

true if the rename succeeded, false otherwise

Throws:

FileDoesNotExistException

InvalidPathException

rename

public boolean rename(TachyonURI srcPath,
             TachyonURI dstPath)
               throws FileDoesNotExistException,
                      InvalidPathException

Rename a file to the given path.

Parameters:

srcPath - The path of the file to rename

dstPath - The new path of the file

Returns:

true if the rename succeeded, false otherwise

Throws:

FileDoesNotExistException

InvalidPathException

reportLostFile

public void reportLostFile(int fileId)

Logs a lost file and sets it to be recovered.

Parameters:

fileId - The id of the file to be recovered

requestFilesInDependency

public void requestFilesInDependency(int depId)

Request that the files for the given dependency be recomputed.

Parameters:

depId - The dependency whose files are to be recomputed

setPinned

public void setPinned(int fileId,
             boolean pinned)
               throws FileDoesNotExistException

Sets the isPinned flag on the given inode and all of its children.

Throws:

FileDoesNotExistException

freepath

public boolean freepath(TachyonURI path,
               boolean recursive)
                 throws TachyonException

Frees files based on the path

Parameters:

path - The file to be freed.

recursive - whether delete the file recursively or not.

Returns:

succeed or not

Throws:

TachyonException

stop

public void stop()

Stops the heartbeat thread.

updateRawTableMetadata

public void updateRawTableMetadata(int tableId,
                          ByteBuffer metadata)
                            throws TableDoesNotExistException,
                                   TachyonException

Update the metadata of a table.

Parameters:

tableId - The id of the table to update

metadata - The new metadata to update the table with

Throws:

TableDoesNotExistException

TachyonException

workerHeartbeat

public Command workerHeartbeat(long workerId,
                      long usedBytes,
                      List<Long> removedBlockIds,
                      Map<Long,List<Long>> addedBlockIds)
                        throws BlockInfoException

The heartbeat of the worker. It updates the information of the worker and removes the given block id's.

Parameters:

workerId - The id of the worker to deal with

usedBytes - The number of bytes used in the worker

removedBlockIds - The list of removed block ids

addedBlockIds - Mapping from id of the StorageDir and id list of blocks evicted in

Returns:

a command specifying an action to take

Throws:

BlockInfoException

writeImage

public void writeImage(com.fasterxml.jackson.databind.ObjectWriter objWriter,
              DataOutputStream dos)
                throws IOException

Create an image of the dependencies and filesystem tree.

Parameters:

objWriter - The used object writer

dos - The target data output stream

Throws:

IOException