spark

Type Members

1. case class Aggregator[K, V, C](createCombiner: (V) ⇒ C, mergeValue: (C, V) ⇒ C, mergeCombiners: (C, C) ⇒ C) extends Product with Serializable

   :: DeveloperApi :: A set of functions used to aggregate data.

2. class ComplexFutureAction[T] extends FutureAction[T]

   A FutureAction for actions that could trigger multiple Spark jobs.

3. abstract class Dependency[T] extends Serializable

   :: DeveloperApi :: Base class for dependencies.

4. case class ExceptionFailure(className: String, description: String, stackTrace: Array[StackTraceElement], fullStackTrace: String, exceptionWrapper: Option[ThrowableSerializationWrapper], accumUpdates: Seq[AccumulableInfo] = collection.this.Seq.empty[Nothing], accums: Seq[AccumulatorV2[_, _]] = immutable.this.Nil) extends TaskFailedReason with Product with Serializable

   :: DeveloperApi :: Task failed due to a runtime exception.

5. case class ExecutorLostFailure(execId: String, exitCausedByApp: Boolean = true, reason: Option[String]) extends TaskFailedReason with Product with Serializable

   :: DeveloperApi :: The task failed because the executor that it was running on was lost.

6. case class FetchFailed(bmAddress: BlockManagerId, shuffleId: Int, mapId: Int, reduceId: Int, message: String) extends TaskFailedReason with Product with Serializable

   :: DeveloperApi :: Task failed to fetch shuffle data from a remote node.

7. trait FutureAction[T] extends Future[T]

   A future for the result of an action to support cancellation.

8. class HashPartitioner extends Partitioner

   A org.apache.spark.Partitioner that implements hash-based partitioning using Java's Object.hashCode.

9. class InterruptibleIterator[+T] extends Iterator[T]

   :: DeveloperApi :: An iterator that wraps around an existing iterator to provide task killing functionality.

10. class JobExecutionStatus extends Enum[JobExecutionStatus]

11. trait JobSubmitter extends AnyRef

    Handle via which a "run" function passed to a ComplexFutureAction can submit jobs for execution.

12. abstract class NarrowDependency[T] extends Dependency[T]

    :: DeveloperApi :: Base class for dependencies where each partition of the child RDD depends on a small number of partitions of the parent RDD.

13. class OneToOneDependency[T] extends NarrowDependency[T]

    :: DeveloperApi :: Represents a one-to-one dependency between partitions of the parent and child RDDs.

14. trait Partition extends Serializable

    An identifier for a partition in an RDD.

15. abstract class Partitioner extends Serializable

    An object that defines how the elements in a key-value pair RDD are partitioned by key.

16. class RangeDependency[T] extends NarrowDependency[T]

    :: DeveloperApi :: Represents a one-to-one dependency between ranges of partitions in the parent and child RDDs.

17. class RangePartitioner[K, V] extends Partitioner

    A org.apache.spark.Partitioner that partitions sortable records by range into roughly equal ranges.

18. class SerializableWritable[T <: Writable] extends Serializable

    Annotations

    @DeveloperApi()

19. class ShuffleDependency[K, V, C] extends Dependency[Product2[K, V]]

    :: DeveloperApi :: Represents a dependency on the output of a shuffle stage.

20. class SimpleFutureAction[T] extends FutureAction[T]

    A FutureAction holding the result of an action that triggers a single job.

21. class SparkConf extends Cloneable with Logging with Serializable

    Configuration for a Spark application.

22. class SparkContext extends Logging

    Main entry point for Spark functionality.

23. class SparkEnv extends Logging

    :: DeveloperApi :: Holds all the runtime environment objects for a running Spark instance (either master or worker), including the serializer, RpcEnv, block manager, map output tracker, etc.

24. class SparkException extends Exception

25. trait SparkExecutorInfo extends Serializable

26. class SparkFirehoseListener extends SparkListenerInterface

27. trait SparkJobInfo extends Serializable

28. trait SparkStageInfo extends Serializable

29. class SparkStatusTracker extends AnyRef

    Low-level status reporting APIs for monitoring job and stage progress.

30. case class TaskCommitDenied(jobID: Int, partitionID: Int, attemptNumber: Int) extends TaskFailedReason with Product with Serializable

    :: DeveloperApi :: Task requested the driver to commit, but was denied.

31. abstract class TaskContext extends Serializable

    Contextual information about a task which can be read or mutated during execution.

32. sealed trait TaskEndReason extends AnyRef

    :: DeveloperApi :: Various possible reasons why a task ended.

33. sealed trait TaskFailedReason extends TaskEndReason

    :: DeveloperApi :: Various possible reasons why a task failed.

34. case class TaskKilled(reason: String) extends TaskFailedReason with Product with Serializable

    :: DeveloperApi :: Task was killed intentionally and needs to be rescheduled.

35. class TaskKilledException extends RuntimeException

    :: DeveloperApi :: Exception thrown when a task is explicitly killed (i.

36. class Accumulable[R, T] extends Serializable

    A data type that can be accumulated, i.

37. trait AccumulableParam[R, T] extends Serializable

    Helper object defining how to accumulate values of a particular type.

38. class Accumulator[T] extends Accumulable[T, T]

    A simpler value of Accumulable where the result type being accumulated is the same as the types of elements being merged, i.

39. trait AccumulatorParam[T] extends AccumulableParam[T, T]

    A simpler version of org.apache.spark.AccumulableParam where the only data type you can add in is the same type as the accumulated value.