SparkContext

Value Members

1. final def !=(arg0: AnyRef): Boolean

   Definition Classes

   AnyRef

2. final def !=(arg0: Any): Boolean

   Definition Classes

   Any

3. final def ##(): Int

   Definition Classes

   AnyRef → Any

4. final def ==(arg0: AnyRef): Boolean

   Definition Classes

   AnyRef

5. final def ==(arg0: Any): Boolean

   Definition Classes

   Any

6. def accumulable[T, R](initialValue: T)(implicit param: AccumulableParam[T, R]): Accumulable[T, R]

   create an accumulatable shared variable, with a += method

   create an accumulatable shared variable, with a += method

   T

   accumulator type

   R

   type that can be added to the accumulator

7. def accumulableCollection[R, T](initialValue: R)(implicit arg0: (R) ⇒ Growable[T] with TraversableOnce[T] with Serializable): Accumulable[R, T]

   create an accumulator from a "mutable collection" type.

   create an accumulator from a "mutable collection" type.

   Growable and TraversableOnce are the standard apis that guarantee += and ++=, implemented by standard mutable collections. So you can use this with mutable Map, Set, etc.

8. def accumulator[T](initialValue: T)(implicit param: AccumulatorParam[T]): Accumulator[T]

9. final def asInstanceOf[T0]: T0

   Definition Classes

   Any

10. def broadcast[T](value: T): Broadcast[T]

11. def clone(): AnyRef

    Attributes

    protected[lang]

    Definition Classes

    AnyRef

    Annotations

    @throws()

12. def defaultMinSplits: Int

13. def defaultParallelism: Int

14. val env: SparkEnv

15. final def eq(arg0: AnyRef): Boolean

    Definition Classes

    AnyRef

16. def equals(arg0: Any): Boolean

    Definition Classes

    AnyRef → Any

17. def finalize(): Unit

    Attributes

    protected[lang]

    Definition Classes

    AnyRef

    Annotations

    @throws()

18. final def getClass(): java.lang.Class[_]

    Definition Classes

    AnyRef → Any

19. def getSparkHome(): Option[String]

20. def hadoopFile[K, V, F <: InputFormat[K, V]](path: String)(implicit km: ClassManifest[K], vm: ClassManifest[V], fm: ClassManifest[F]): RDD[(K, V)]

21. def hadoopFile[K, V, F <: InputFormat[K, V]](path: String, minSplits: Int)(implicit km: ClassManifest[K], vm: ClassManifest[V], fm: ClassManifest[F]): RDD[(K, V)]

    Smarter version of hadoopFile() that uses class manifests to figure out the classes of keys, values and the InputFormat so that users don't need to pass them directly.

22. def hadoopFile[K, V](path: String, inputFormatClass: Class[_ <: org.apache.hadoop.mapred.InputFormat[K,V]], keyClass: Class[K], valueClass: Class[V], minSplits: Int): RDD[(K, V)]

    Get an RDD for a Hadoop file with an arbitrary InputFormat

23. def hadoopRDD[K, V](conf: JobConf, inputFormatClass: Class[_ <: org.apache.hadoop.mapred.InputFormat[K,V]], keyClass: Class[K], valueClass: Class[V], minSplits: Int): RDD[(K, V)]

    Get an RDD for a Hadoop-readable dataset from a Hadooop JobConf giving its InputFormat and any other necessary info (e.

    Get an RDD for a Hadoop-readable dataset from a Hadooop JobConf giving its InputFormat and any other necessary info (e.g. file name for a filesystem-based dataset, table name for HyperTable, etc).

24. def hashCode(): Int

    Definition Classes

    AnyRef → Any

25. def initLogging(): Unit

    Definition Classes

    Logging

26. final def isInstanceOf[T0]: Boolean

    Definition Classes

    Any

27. val jars: Seq[String]

28. def log: Logger

    Definition Classes

    Logging

29. def logDebug(msg: ⇒ String, throwable: Throwable): Unit

    Definition Classes

    Logging

30. def logDebug(msg: ⇒ String): Unit

    Definition Classes

    Logging

31. def logError(msg: ⇒ String, throwable: Throwable): Unit

    Definition Classes

    Logging

32. def logError(msg: ⇒ String): Unit

    Definition Classes

    Logging

33. def logInfo(msg: ⇒ String, throwable: Throwable): Unit

    Definition Classes

    Logging

34. def logInfo(msg: ⇒ String): Unit

    Definition Classes

    Logging

35. def logWarning(msg: ⇒ String, throwable: Throwable): Unit

    Definition Classes

    Logging

36. def logWarning(msg: ⇒ String): Unit

    Definition Classes

    Logging

37. def makeRDD[T](seq: Seq[T], numSlices: Int)(implicit arg0: ClassManifest[T]): RDD[T]

38. final def ne(arg0: AnyRef): Boolean

    Definition Classes

    AnyRef

39. def newAPIHadoopFile[K, V, F <: InputFormat[K, V]](path: String, fClass: Class[F], kClass: Class[K], vClass: Class[V], conf: Configuration): RDD[(K, V)]

    Get an RDD for a given Hadoop file with an arbitrary new API InputFormat and extra configuration options to pass to the input format.

40. def newAPIHadoopFile[K, V, F <: InputFormat[K, V]](path: String)(implicit km: ClassManifest[K], vm: ClassManifest[V], fm: ClassManifest[F]): RDD[(K, V)]

    Get an RDD for a Hadoop file with an arbitrary new API InputFormat.

41. def newAPIHadoopRDD[K, V, F <: InputFormat[K, V]](conf: Configuration, fClass: Class[F], kClass: Class[K], vClass: Class[V]): RDD[(K, V)]

    Get an RDD for a given Hadoop file with an arbitrary new API InputFormat and extra configuration options to pass to the input format.

42. final def notify(): Unit

    Definition Classes

    AnyRef

43. final def notifyAll(): Unit

    Definition Classes

    AnyRef

44. def objectFile[T](path: String, minSplits: Int = defaultMinSplits)(implicit arg0: ClassManifest[T]): RDD[T]

    Load an RDD saved as a SequenceFile containing serialized objects, with NullWritable keys and BytesWritable values that contain a serialized partition.

    Load an RDD saved as a SequenceFile containing serialized objects, with NullWritable keys and BytesWritable values that contain a serialized partition. This is still an experimental storage format and may not be supported exactly as is in future Spark releases. It will also be pretty slow if you use the default serializer (Java serialization), though the nice thing about it is that there's very little effort required to save arbitrary objects.

45. def parallelize[T](seq: Seq[T], numSlices: Int)(implicit arg0: ClassManifest[T]): RDD[T]

46. def runJob[T, U](rdd: RDD[T], func: (Iterator[T]) ⇒ U)(implicit arg0: ClassManifest[U]): Array[U]

47. def runJob[T, U](rdd: RDD[T], func: (TaskContext, Iterator[T]) ⇒ U)(implicit arg0: ClassManifest[U]): Array[U]

    Run a job on all partitions in an RDD and return the results in an array.

48. def runJob[T, U](rdd: RDD[T], func: (Iterator[T]) ⇒ U, partitions: Seq[Int], allowLocal: Boolean)(implicit arg0: ClassManifest[U]): Array[U]

49. def runJob[T, U](rdd: RDD[T], func: (TaskContext, Iterator[T]) ⇒ U, partitions: Seq[Int], allowLocal: Boolean)(implicit arg0: ClassManifest[U]): Array[U]

    Run a function on a given set of partitions in an RDD and return the results.

    Run a function on a given set of partitions in an RDD and return the results. This is the main entry point to the scheduler, by which all actions get launched. The allowLocal flag specifies whether the scheduler can run the computation on the master rather than shipping it out to the cluster, for short actions like first().

50. def sequenceFile[K, V](path: String, minSplits: Int = defaultMinSplits)(implicit km: ClassManifest[K], vm: ClassManifest[V], kcf: () ⇒ WritableConverter[K], vcf: () ⇒ WritableConverter[V]): RDD[(K, V)]

    Version of sequenceFile() for types implicitly convertible to Writables through a WritableConverter.

    Version of sequenceFile() for types implicitly convertible to Writables through a WritableConverter.

    WritableConverters are provided in a somewhat strange way (by an implicit function) to support both subclasses of Writable and types for which we define a converter (e.g. Int to IntWritable). The most natural thing would've been to have implicit objects for the converters, but then we couldn't have an object for every subclass of Writable (you can't have a parameterized singleton object). We use functions instead to create a new converter for the appropriate type. In addition, we pass the converter a ClassManifest of its type to allow it to figure out the Writable class to use in the subclass case.

51. def sequenceFile[K, V](path: String, keyClass: Class[K], valueClass: Class[V]): RDD[(K, V)]

52. def sequenceFile[K, V](path: String, keyClass: Class[K], valueClass: Class[V], minSplits: Int): RDD[(K, V)]

    Get an RDD for a Hadoop SequenceFile with given key and value types

53. val sparkHome: String

54. def stop(): Unit

55. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes

    AnyRef

56. def textFile(path: String, minSplits: Int = defaultMinSplits): RDD[String]

57. def toString(): String

    Definition Classes

    AnyRef → Any

58. def union[T](rdds: RDD[T]*)(implicit arg0: ClassManifest[T]): RDD[T]

    Build the union of a list of RDDs.

59. final def wait(): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws()

60. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws()

61. final def wait(arg0: Long): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws()

62. def waitForRegister(): Unit