FSCleanUp

Instance Constructors

new FSCleanUp(baseFolder: String, toRemove: CleanUpStrategy[FileStatus], labelsToClean: Seq[String])

baseFolder
root folder that contains label folders
toRemove
returns list of snapshot/folder to remove

Value Members

final def !=(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def ##(): Int

Definition Classes
AnyRef → Any
final def ==(arg0: Any): Boolean

Definition Classes
AnyRef → Any
val actionName: String

For representing the action
For representing the action

Definition Classes
FSCleanUp → DataFlowAction
final def asInstanceOf[T0]: T0

Definition Classes
Any
val baseFolder: String

root folder that contains label folders
def clone(): AnyRef

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( ... )
def description: String

Definition Classes
DataFlowAction
final def eq(arg0: AnyRef): Boolean

Definition Classes
AnyRef
def finalize(): Unit

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( classOf[java.lang.Throwable] )
def flowState(inputs: DataFlowEntities): DataFlowActionState

Action has the responsibility of assessing itself and produce DataFlowActionState, that will be used by the executors to determine if they can call performAction or not.
Action has the responsibility of assessing itself and produce DataFlowActionState, that will be used by the executors to determine if they can call performAction or not. Also can be used for progress monitoring. This will allow for more custom actions without modifying the executors
inputs
- action will study the state of the inputs in order to generate self assessment
returns
- an instance of the DataFlowActionState

Definition Classes
DataFlowAction
final def getClass(): Class[_]

Definition Classes
AnyRef → Any
val guid: String

Unique id of the action, but using it for adding behaviours can be problematic due to Interceptors that are defined at a much later stage.
Unique id of the action, but using it for adding behaviours can be problematic due to Interceptors that are defined at a much later stage. Because of that ActionSchedulers must NOT use this guid.

Definition Classes
DataFlowAction
val inputLabels: List[String]

The unique identifiers for the inputs to this action
The unique identifiers for the inputs to this action

Definition Classes
FSCleanUp → DataFlowAction
final def isInstanceOf[T0]: Boolean

Definition Classes
Any
def isTraceEnabled(): Boolean

Attributes
protected
Definition Classes
Logging
val labelsToClean: Seq[String]
def logAndReturn[A](a: A, msg: String, level: Level): A

Takes a value of type A and a msg to log, returning a and logging the message at the desired level
Takes a value of type A and a msg to log, returning a and logging the message at the desired level
returns
a

Definition Classes
Logging
def logAndReturn[A](a: A, message: (A) ⇒ String, level: Level): A

Takes a value of type A and a function message from A to String, logs the value of invoking message(a) at the level described by the level parameter
Takes a value of type A and a function message from A to String, logs the value of invoking message(a) at the level described by the level parameter
returns
a
Definition Classes
Logging
Example:
1. logAndReturn(1, (num: Int) => s"number: $num", Info) // In the log we would see a log corresponding to "number 1"
def logDebug(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logDebug(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
def logError(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logError(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
def logInfo(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logInfo(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
def logLabel: String

Definition Classes
DataFlowAction
def logName: String

Attributes
protected
Definition Classes
Logging
def logTrace(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logTrace(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
def logWarning(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logWarning(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
final def ne(arg0: AnyRef): Boolean

Definition Classes
AnyRef
final def notify(): Unit

Definition Classes
AnyRef
final def notifyAll(): Unit

Definition Classes
AnyRef
val outputLabels: List[String]

The unique identifiers for the outputs to this action
The unique identifiers for the outputs to this action

Definition Classes
FSCleanUp → DataFlowAction
def performAction(inputs: DataFlowEntities, flowContext: SparkFlowContext): Try[ActionResult]

Perform the action
Perform the action
inputs
the DataFlowEntities corresponding to the inputLabels
flowContext
context of the flow in which this action runs
returns
the action outputs (these must be declared in the same order as their labels in outputLabels)

Definition Classes
FSCleanUp → SparkDataFlowAction
def performAction[C <: FlowContext](inputs: DataFlowEntities, flowContext: C): Try[ActionResult]

Perform the action
Perform the action
inputs
the DataFlowEntities corresponding to the inputLabels
flowContext
context of the flow in which this action runs
returns
the action outputs (these must be declared in the same order as their labels in outputLabels)

Definition Classes
SparkDataFlowAction → DataFlowAction
def requiresAllInputs: Boolean

This action can only be executed if all of the inputs are not empty.
This action can only be executed if all of the inputs are not empty. An input can be explicitly marked as empty. If false, than one or more inputs can be empty to start execution.

Definition Classes
DataFlowAction
def schedulingGuid: String

Interceptors must not override this property, as certain behaviours of the data flow (Ex, execution pools) will be associated with this scheduling guid.
Interceptors must not override this property, as certain behaviours of the data flow (Ex, execution pools) will be associated with this scheduling guid. Also ActionScheduler will use this guid to track scheduled actions.

Definition Classes
DataFlowAction
final def synchronized[T0](arg0: ⇒ T0): T0

Definition Classes
AnyRef
val toRemove: CleanUpStrategy[FileStatus]

returns list of snapshot/folder to remove
final def wait(): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long, arg1: Int): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )

Related Doc: package spark

case class FSCleanUp(baseFolder: String, toRemove: CleanUpStrategy[FileStatus], labelsToClean: Seq[String]) extends SparkDataFlowAction with Logging with Product with Serializable

Instance Constructors

new FSCleanUp(baseFolder: String, toRemove: CleanUpStrategy[FileStatus], labelsToClean: Seq[String])

Value Members

final def !=(arg0: Any): Boolean

final def ##(): Int

final def ==(arg0: Any): Boolean

val actionName: String

final def asInstanceOf[T0]: T0

val baseFolder: String

def clone(): AnyRef

def description: String

final def eq(arg0: AnyRef): Boolean

def finalize(): Unit

def flowState(inputs: DataFlowEntities): DataFlowActionState

final def getClass(): Class[_]

val guid: String

val inputLabels: List[String]

final def isInstanceOf[T0]: Boolean

def isTraceEnabled(): Boolean

val labelsToClean: Seq[String]

def logAndReturn[A](a: A, msg: String, level: Level): A

def logAndReturn[A](a: A, message: (A) ⇒ String, level: Level): A

def logDebug(msg: ⇒ String, throwable: Throwable): Unit

def logDebug(msg: ⇒ String): Unit

def logError(msg: ⇒ String, throwable: Throwable): Unit

def logError(msg: ⇒ String): Unit

def logInfo(msg: ⇒ String, throwable: Throwable): Unit

def logInfo(msg: ⇒ String): Unit

def logLabel: String

def logName: String

def logTrace(msg: ⇒ String, throwable: Throwable): Unit

def logTrace(msg: ⇒ String): Unit

def logWarning(msg: ⇒ String, throwable: Throwable): Unit

def logWarning(msg: ⇒ String): Unit

final def ne(arg0: AnyRef): Boolean

final def notify(): Unit

final def notifyAll(): Unit

val outputLabels: List[String]

def performAction(inputs: DataFlowEntities, flowContext: SparkFlowContext): Try[ActionResult]

def performAction[C <: FlowContext](inputs: DataFlowEntities, flowContext: C): Try[ActionResult]

def requiresAllInputs: Boolean

def schedulingGuid: String

final def synchronized[T0](arg0: ⇒ T0): T0

val toRemove: CleanUpStrategy[FileStatus]

final def wait(): Unit

final def wait(arg0: Long, arg1: Int): Unit

final def wait(arg0: Long): Unit

Inherited from Serializable

Inherited from Serializable

Inherited from Product

Inherited from Equals

Inherited from Logging

Inherited from SparkDataFlowAction

Inherited from DataFlowAction

Inherited from AnyRef

Inherited from Any

Ungrouped