org.apache.flink.runtime.state.filesystem

Class FsStateBackend

java.lang.Object

org.apache.flink.runtime.state.StateBackend<FsStateBackend>

org.apache.flink.runtime.state.filesystem.FsStateBackend

All Implemented Interfaces:

Serializable

public class FsStateBackend
extends StateBackend<FsStateBackend>

The file state backend is a state backend that stores the state of streaming jobs in a file system.

The state backend has one core directory into which it puts all checkpoint data. Inside that directory, it creates a directory per job, inside which each checkpoint gets a directory, with files for each state, for example: hdfs://namenode:port/flink-checkpoints/<job-id>/chk-17/6ba7b810-9dad-11d1-80b4-00c04fd430c8

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	FsStateBackend.FsCheckpointStateOutputStream A CheckpointStateOutputStream that writes into a file and returns the path to that file upon closing.

Nested classes/interfaces inherited from class org.apache.flink.runtime.state.StateBackend

StateBackend.CheckpointStateOutputStream, StateBackend.CheckpointStateOutputView

Constructor Summary

Constructors

Constructor and Description
FsStateBackend(Path checkpointDataUri) Creates a new state backend that stores its checkpoint data in the file system and location defined by the given URI.
FsStateBackend(String checkpointDataUri) Creates a new state backend that stores its checkpoint data in the file system and location defined by the given URI.
FsStateBackend(URI checkpointDataUri) Creates a new state backend that stores its checkpoint data in the file system and location defined by the given URI.

Method Summary

Methods

Modifier and Type	Method and Description
<S extends Serializable> StateHandle<S>	checkpointStateSerializable(S state, long checkpointID, long timestamp) Writes the given state into the checkpoint, and returns a handle that can retrieve the state back.
void	close() Closes the state backend, releasing all internal resources, but does not delete any persistent checkpoint data.
FsStateBackend.FsCheckpointStateOutputStream	createCheckpointStateOutputStream(long checkpointID, long timestamp) Creates an output stream that writes into the state of the given checkpoint.
<K,V> FsHeapKvState<K,V>	createKvState(TypeSerializer<K> keySerializer, TypeSerializer<V> valueSerializer, V defaultValue) Creates a key/value state backed by this state backend.
void	disposeAllStateForCurrentJob() Disposes all state associated with the current job.
Path	getBasePath() Gets the base directory where all state-containing files are stored.
Path	getCheckpointDirectory() Gets the directory where this state backend stores its checkpoint data.
FileSystem	getFileSystem() Gets the file system handle for the file system that stores the state for this backend.
void	initializeForJob(JobID jobId) This method is called by the task upon deployment to initialize the state backend for data for a specific job.
boolean	isInitialized() Checks whether this state backend is initialized.
String	toString()

Methods inherited from class org.apache.flink.runtime.state.StateBackend

createCheckpointStateOutputView

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

FsStateBackend

public FsStateBackend(String checkpointDataUri)
               throws IOException

Creates a new state backend that stores its checkpoint data in the file system and location defined by the given URI.

A file system for the file system scheme in the URI (e.g., 'file://', 'hdfs://', or 'S3://') must be accessible via FileSystem.get(URI).

For a state backend targeting HDFS, this means that the URI must either specify the authority (host and port), or that the Hadoop configuration that describes that information must be in the classpath.

Parameters:

checkpointDataUri - The URI describing the filesystem (scheme and optionally authority), and the path to teh checkpoint data directory.

Throws:

IOException - Thrown, if no file system can be found for the scheme in the URI.

FsStateBackend

public FsStateBackend(Path checkpointDataUri)
               throws IOException

Creates a new state backend that stores its checkpoint data in the file system and location defined by the given URI.

A file system for the file system scheme in the URI (e.g., 'file://', 'hdfs://', or 'S3://') must be accessible via FileSystem.get(URI).

For a state backend targeting HDFS, this means that the URI must either specify the authority (host and port), or that the Hadoop configuration that describes that information must be in the classpath.

Parameters:

checkpointDataUri - The URI describing the filesystem (scheme and optionally authority), and the path to teh checkpoint data directory.

Throws:

IOException - Thrown, if no file system can be found for the scheme in the URI.

FsStateBackend

public FsStateBackend(URI checkpointDataUri)
               throws IOException

Creates a new state backend that stores its checkpoint data in the file system and location defined by the given URI.

A file system for the file system scheme in the URI (e.g., 'file://', 'hdfs://', or 'S3://') must be accessible via FileSystem.get(URI).

For a state backend targeting HDFS, this means that the URI must either specify the authority (host and port), or that the Hadoop configuration that describes that information must be in the classpath.

Parameters:

checkpointDataUri - The URI describing the filesystem (scheme and optionally authority), and the path to teh checkpoint data directory.

Throws:

IOException - Thrown, if no file system can be found for the scheme in the URI.

Method Detail

getBasePath

public Path getBasePath()

Gets the base directory where all state-containing files are stored. The job specific directory is created inside this directory.

Returns:

The base directory.

getCheckpointDirectory

public Path getCheckpointDirectory()

Gets the directory where this state backend stores its checkpoint data. Will be null if the state backend has not been initialized.

Returns:

The directory where this state backend stores its checkpoint data.

isInitialized

public boolean isInitialized()

Checks whether this state backend is initialized. Note that initialization does not carry across serialization. After each serialization, the state backend needs to be initialized.

Returns:

True, if the file state backend has been initialized, false otherwise.

getFileSystem

public FileSystem getFileSystem()

Gets the file system handle for the file system that stores the state for this backend.

Returns:

This backend's file system handle.

initializeForJob

public void initializeForJob(JobID jobId)
                      throws Exception

Description copied from class: StateBackend

This method is called by the task upon deployment to initialize the state backend for data for a specific job.

Specified by:

initializeForJob in class StateBackend<FsStateBackend>

Parameters:

jobId - The ID of the job for which the state backend instance checkpoints data.

Throws:

Exception - Overwritten versions of this method may throw exceptions, in which case the job that uses the state backend is considered failed during deployment.

disposeAllStateForCurrentJob

public void disposeAllStateForCurrentJob()
                                  throws Exception

Description copied from class: StateBackend

Disposes all state associated with the current job.

Specified by:

disposeAllStateForCurrentJob in class StateBackend<FsStateBackend>

Throws:

Exception - Exceptions may occur during disposal of the state and should be forwarded.

close

public void close()
           throws Exception

Description copied from class: StateBackend

Closes the state backend, releasing all internal resources, but does not delete any persistent checkpoint data.

Specified by:

close in class StateBackend<FsStateBackend>

Throws:

Exception - Exceptions can be forwarded and will be logged by the system

createKvState

public <K,V> FsHeapKvState<K,V> createKvState(TypeSerializer<K> keySerializer,
                                     TypeSerializer<V> valueSerializer,
                                     V defaultValue)
                                 throws Exception

Description copied from class: StateBackend

Creates a key/value state backed by this state backend.

Specified by:

createKvState in class StateBackend<FsStateBackend>

Type Parameters:

K - The type of the key.

V - The type of the value.

Parameters:

keySerializer - The serializer for the key.

valueSerializer - The serializer for the value.

defaultValue - The value that is returned when no other value has been associated with a key, yet.

Returns:

A new key/value state backed by this backend.

Throws:

Exception - Exceptions may occur during initialization of the state and should be forwarded.

checkpointStateSerializable

public <S extends Serializable> StateHandle<S> checkpointStateSerializable(S state,
                                                                  long checkpointID,
                                                                  long timestamp)
                                                                throws Exception

Description copied from class: StateBackend

Writes the given state into the checkpoint, and returns a handle that can retrieve the state back.

Specified by:

checkpointStateSerializable in class StateBackend<FsStateBackend>

Type Parameters:

S - The type of the state.

Parameters:

state - The state to be checkpointed.

checkpointID - The ID of the checkpoint.

timestamp - The timestamp of the checkpoint.

Returns:

A state handle that can retrieve the checkpoined state.

Throws:

Exception - Exceptions may occur during serialization / storing the state and should be forwarded.

createCheckpointStateOutputStream

public FsStateBackend.FsCheckpointStateOutputStream createCheckpointStateOutputStream(long checkpointID,
                                                                             long timestamp)
                                                                               throws Exception

Description copied from class: StateBackend

Creates an output stream that writes into the state of the given checkpoint. When the stream is closes, it returns a state handle that can retrieve the state back.

Specified by:

createCheckpointStateOutputStream in class StateBackend<FsStateBackend>

Parameters:

checkpointID - The ID of the checkpoint.

timestamp - The timestamp of the checkpoint.

Returns:

An output stream that writes state for the given checkpoint.

Throws:

Exception - Exceptions may occur while creating the stream and should be forwarded.

toString

public String toString()

Overrides:

toString in class Object