spark

ShuffledRDD

class ShuffledRDD[K, V, C] extends RDD[(K, C)]

Linear Supertypes
RDD[(K, C)], Serializable, Serializable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. Hide All
  2. Show all
  1. ShuffledRDD
  2. RDD
  3. Serializable
  4. Serializable
  5. AnyRef
  6. Any
Visibility
  1. Public
  2. All

Instance Constructors

  1. new ShuffledRDD(parent: RDD[(K, V)], aggregator: Aggregator[K, V, C], part: Partitioner)

Value Members

  1. final def !=(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  2. final def !=(arg0: Any): Boolean

    Definition Classes
    Any
  3. final def ##(): Int

    Definition Classes
    AnyRef → Any
  4. def ++(other: RDD[(K, C)]): RDD[(K, C)]

    Definition Classes
    RDD
  5. final def ==(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  6. final def ==(arg0: Any): Boolean

    Definition Classes
    Any
  7. def aggregate[U](zeroValue: U)(seqOp: (U, (K, C)) ⇒ U, combOp: (U, U) ⇒ U)(implicit arg0: ClassManifest[U]): U

    Aggregate the elements of each partition, and then the results for all the partitions, using given combine functions and a neutral "zero value".

    Aggregate the elements of each partition, and then the results for all the partitions, using given combine functions and a neutral "zero value". This function can return a different result type, U, than the type of this RDD, T. Thus, we need one operation for merging a T into an U and one operation for merging two U's, as in scala.TraversableOnce. Both of these functions are allowed to modify and return their first argument instead of creating a new U to avoid memory allocation.

    Definition Classes
    RDD
  8. final def asInstanceOf[T0]: T0

    Definition Classes
    Any
  9. def cache(): RDD[(K, C)]

    Definition Classes
    RDD
  10. def cartesian[U](other: RDD[U])(implicit arg0: ClassManifest[U]): RDD[((K, C), U)]

    Definition Classes
    RDD
  11. def clone(): AnyRef

    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws()
  12. def collect(): Array[(K, C)]

    Definition Classes
    RDD
  13. def compute(split: Split): Iterator[(K, C)]

    Definition Classes
    ShuffledRDDRDD
  14. def context: SparkContext

    Definition Classes
    RDD
  15. def count(): Long

    Definition Classes
    RDD
  16. val dep: ShuffleDependency[K, V, C]

  17. val dependencies: List[ShuffleDependency[K, V, C]]

    Definition Classes
    ShuffledRDDRDD
  18. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  19. def equals(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any
  20. def filter(f: ((K, C)) ⇒ Boolean): RDD[(K, C)]

    Definition Classes
    RDD
  21. def finalize(): Unit

    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws()
  22. def first(): (K, C)

    Definition Classes
    RDD
  23. def flatMap[U](f: ((K, C)) ⇒ TraversableOnce[U])(implicit arg0: ClassManifest[U]): RDD[U]

    Definition Classes
    RDD
  24. def fold(zeroValue: (K, C))(op: ((K, C), (K, C)) ⇒ (K, C)): (K, C)

    Aggregate the elements of each partition, and then the results for all the partitions, using a given associative function and a neutral "zero value".

    Aggregate the elements of each partition, and then the results for all the partitions, using a given associative function and a neutral "zero value". The function op(t1, t2) is allowed to modify t1 and return it as its result value to avoid object allocation; however, it should not modify t2.

    Definition Classes
    RDD
  25. def foreach(f: ((K, C)) ⇒ Unit): Unit

    Definition Classes
    RDD
  26. final def getClass(): java.lang.Class[_]

    Definition Classes
    AnyRef → Any
  27. def glom(): RDD[Array[(K, C)]]

    Definition Classes
    RDD
  28. def groupBy[K](f: ((K, C)) ⇒ K)(implicit arg0: ClassManifest[K]): RDD[(K, Seq[(K, C)])]

    Definition Classes
    RDD
  29. def groupBy[K](f: ((K, C)) ⇒ K, numSplits: Int)(implicit arg0: ClassManifest[K]): RDD[(K, Seq[(K, C)])]

    Definition Classes
    RDD
  30. def hashCode(): Int

    Definition Classes
    AnyRef → Any
  31. val id: Int

    Definition Classes
    RDD
  32. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any
  33. final def iterator(split: Split): Iterator[(K, C)]

    Definition Classes
    RDD
  34. def map[U](f: ((K, C)) ⇒ U)(implicit arg0: ClassManifest[U]): RDD[U]

    Definition Classes
    RDD
  35. def mapPartitions[U](f: (Iterator[(K, C)]) ⇒ Iterator[U])(implicit arg0: ClassManifest[U]): RDD[U]

    Definition Classes
    RDD
  36. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  37. final def notify(): Unit

    Definition Classes
    AnyRef
  38. final def notifyAll(): Unit

    Definition Classes
    AnyRef
  39. val partitioner: Some[Partitioner]

    Definition Classes
    ShuffledRDDRDD
  40. def pipe(command: Seq[String], env: Map[String, String]): RDD[String]

    Definition Classes
    RDD
  41. def pipe(command: Seq[String]): RDD[String]

    Definition Classes
    RDD
  42. def pipe(command: String): RDD[String]

    Definition Classes
    RDD
  43. def preferredLocations(split: Split): scala.collection.immutable.Nil.type

    Definition Classes
    ShuffledRDDRDD
  44. def reduce(f: ((K, C), (K, C)) ⇒ (K, C)): (K, C)

    Definition Classes
    RDD
  45. def sample(withReplacement: Boolean, fraction: Double, seed: Int): RDD[(K, C)]

    Definition Classes
    RDD
  46. def saveAsObjectFile(path: String): Unit

    Definition Classes
    RDD
  47. def saveAsTextFile(path: String): Unit

    Definition Classes
    RDD
  48. def splits: Array[Split]

    Definition Classes
    ShuffledRDDRDD
  49. val splits_: Array[Split]

  50. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef
  51. def take(num: Int): Array[(K, C)]

    Take the first num elements of the RDD.

    Take the first num elements of the RDD. This currently scans the partitions *one by one*, so it will be slow if a lot of partitions are required. In that case, use collect() to get the whole RDD instead.

    Definition Classes
    RDD
  52. def takeSample(withReplacement: Boolean, num: Int, seed: Int): Array[(K, C)]

    Definition Classes
    RDD
  53. def toArray(): Array[(K, C)]

    Definition Classes
    RDD
  54. def toString(): String

    Definition Classes
    AnyRef → Any
  55. def union(other: RDD[(K, C)]): RDD[(K, C)]

    Definition Classes
    RDD
  56. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws()
  57. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws()
  58. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws()

Inherited from RDD[(K, C)]

Inherited from Serializable

Inherited from Serializable

Inherited from AnyRef

Inherited from Any