_NamedRdds

Abstract Value Members

abstract def defaultTimeout: Timeout
abstract def destroy(name: String): Unit

Destroys an RDD with the given name, if one existed.
Destroys an RDD with the given name, if one existed. Has no effect if no RDD with this name exists.
name
the unique name of the RDD. The uniqueness is scoped to the current SparkContext.
abstract def get[T](name: String)(implicit timeout: Timeout = defaultTimeout): Option[RDD[T]]

Gets an RDD with the given name if it already exists and is cached by the RddManager.
Gets an RDD with the given name if it already exists and is cached by the RddManager. If the RDD does not exist, None is returned.
Note that a previously-known RDD could 'disappear' if it hasn't been used for a while, because the SparkContext garbage-collects old cached RDDs.
T
the generic type of the RDD.
name
the unique name of the RDD. The uniqueness is scoped to the current SparkContext.
timeout
if the RddManager doesn't respond within this timeout, an error will be thrown.
returns
the RDD with the given name.

Exceptions thrown
java.util.concurrent.TimeoutException if the request to the RddManager times out.
abstract def getNames(): Iterable[String]

Returns the names of all named RDDs that are managed by the RddManager.
Returns the names of all named RDDs that are managed by the RddManager.
Note: this returns a snapshot of RDD names at one point in time. The caller should always expect that the data returned from this method may be stale and incorrect.
returns
a collection of RDD names representing RDDs managed by the RddManager.
abstract def getOrElseCreate[T](name: String, rddGen: ⇒ RDD[T], forceComputation: Boolean = true, storageLevel: StorageLevel = defaultStorageLevel)(implicit timeout: Timeout = defaultTimeout): RDD[T]

Gets an RDD with the given name, or creates it if one doesn't already exist.
Gets an RDD with the given name, or creates it if one doesn't already exist.
If the given RDD has already been computed by another job and cached in memory, this method will return a reference to the cached RDD. If the RDD has never been computed, then the generator will be called to compute it, in the caller's thread, and the result will be cached and returned to the caller.
If an RDD is requested by thread B while thread A is generating the RDD, thread B will block up to the duration specified by @timeout. If thread A finishes generating the RDD within that time, then thread B will get a reference to the newly-created RDD. If thread A does not finish generating the RDD within that time, then thread B will throw a timeout exception.
T
the generic type of the RDD.
name
the unique name of the RDD. The uniqueness is scoped to the current SparkContext.
rddGen
a 0-ary function which will generate the RDD if it doesn't already exist.
forceComputation
if true, forces the RDD to be computed by calling count().
storageLevel
the storage level to persist the RDD with. Default: StorageLevel.MEMORY_ONLY.
timeout
if the RddManager doesn't respond within this timeout, an error will be thrown.
returns
the RDD with the given name.

Exceptions thrown
java.lang.RuntimeException wrapping any error that occurs within the generator function.
java.util.concurrent.TimeoutException if the request to the RddManager times out.
abstract def update[T](name: String, rddGen: ⇒ RDD[T], forceComputation: Boolean = true, storageLevel: StorageLevel = defaultStorageLevel)(implicit timeout: Timeout = defaultTimeout): RDD[T]

Replaces an existing RDD with a given name with a new RDD.
Replaces an existing RDD with a given name with a new RDD. If an old RDD for the given name existed, it is un-persisted (non-blocking) and destroyed. It is safe to call this method when there is no existing RDD with the given name. If multiple threads call this around the same time, the end result is undefined - one of the generated RDDs will win and will be returned from future calls to get().
The rdd generator function will be called from the caller's thread. Note that if this is called at the same time as getOrElseCreate() for the same name, and completes before the getOrElseCreate() call, then threads waiting for the result of getOrElseCreate() will unblock with the result of this update() call. When the getOrElseCreate() succeeds, it will replace the result of this update() call.
T
the generic type of the RDD.
name
the unique name of the RDD. The uniqueness is scoped to the current SparkContext.
rddGen
a 0-ary function which will be called to generate the RDD in the caller's thread.
forceComputation
if true, forces the RDD to be computed by calling count().
storageLevel
the storage level to persist the RDD with. Default: StorageLevel.MEMORY_ONLY.
returns
the RDD with the given name.

Concrete Value Members

final def !=(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def ##(): Int

Definition Classes
AnyRef → Any
final def ==(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def asInstanceOf[T0]: T0

Definition Classes
Any
def clone(): AnyRef

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( ... )
val defaultStorageLevel: StorageLevel
final def eq(arg0: AnyRef): Boolean

Definition Classes
AnyRef
def equals(arg0: Any): Boolean

Definition Classes
AnyRef → Any
def finalize(): Unit

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( classOf[java.lang.Throwable] )
final def getClass(): Class[_]

Definition Classes
AnyRef → Any
def hashCode(): Int

Definition Classes
AnyRef → Any
final def isInstanceOf[T0]: Boolean

Definition Classes
Any
final def ne(arg0: AnyRef): Boolean

Definition Classes
AnyRef
final def notify(): Unit

Definition Classes
AnyRef
final def notifyAll(): Unit

Definition Classes
AnyRef
final def synchronized[T0](arg0: ⇒ T0): T0

Definition Classes
AnyRef
def toString(): String

Definition Classes
AnyRef → Any
final def wait(): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long, arg1: Int): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )

Related Doc: package NamedRddSupport

trait _NamedRdds extends AnyRef

Abstract Value Members

abstract def defaultTimeout: Timeout

abstract def destroy(name: String): Unit

abstract def get[T](name: String)(implicit timeout: Timeout = defaultTimeout): Option[RDD[T]]

abstract def getNames(): Iterable[String]

abstract def getOrElseCreate[T](name: String, rddGen: ⇒ RDD[T], forceComputation: Boolean = true, storageLevel: StorageLevel = defaultStorageLevel)(implicit timeout: Timeout = defaultTimeout): RDD[T]

abstract def update[T](name: String, rddGen: ⇒ RDD[T], forceComputation: Boolean = true, storageLevel: StorageLevel = defaultStorageLevel)(implicit timeout: Timeout = defaultTimeout): RDD[T]

Concrete Value Members

final def !=(arg0: Any): Boolean

final def ##(): Int

final def ==(arg0: Any): Boolean

final def asInstanceOf[T0]: T0

def clone(): AnyRef

val defaultStorageLevel: StorageLevel

final def eq(arg0: AnyRef): Boolean

def equals(arg0: Any): Boolean

def finalize(): Unit

final def getClass(): Class[_]

def hashCode(): Int

final def isInstanceOf[T0]: Boolean

final def ne(arg0: AnyRef): Boolean

final def notify(): Unit

final def notifyAll(): Unit

final def synchronized[T0](arg0: ⇒ T0): T0

def toString(): String

final def wait(): Unit

final def wait(arg0: Long, arg1: Int): Unit

final def wait(arg0: Long): Unit

Inherited from AnyRef

Inherited from Any

Ungrouped