Trait/Object

com.twitter.scalding

Config

Related Docs: object Config | package scalding

Permalink

trait Config extends Serializable

This is a wrapper class on top of Map[String, String]

Linear Supertypes
Serializable, Serializable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. Config
  2. Serializable
  3. Serializable
  4. AnyRef
  5. Any
  1. Hide All
  2. Show all
Visibility
  1. Public
  2. All

Abstract Value Members

  1. abstract def toMap: Map[String, String]

    Permalink

Concrete Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  3. def +(kv: (String, String)): Config

    Permalink
  4. def ++(that: Config): Config

    Permalink
  5. def -(k: String): Config

    Permalink
  6. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  7. def addCascadingClassSerializationTokens(clazzes: Set[Class[_]]): Config

    Permalink

    The serialization of your data will be smaller if any classes passed between tasks in your job are listed here.

    The serialization of your data will be smaller if any classes passed between tasks in your job are listed here. Without this, strings are used to write the types IN EACH RECORD, which compression probably takes care of, but compression acts AFTER the data is serialized into buffers and spilling has been triggered.

  8. def addReducerEstimator(clsName: String): Config

    Permalink

    Prepend an estimator so it will be tried first.

    Prepend an estimator so it will be tried first. If it returns None, the previously-set estimators will be tried in order.

  9. def addReducerEstimator[T](cls: Class[T]): Config

    Permalink

    Prepend an estimator so it will be tried first.

    Prepend an estimator so it will be tried first. If it returns None, the previously-set estimators will be tried in order.

  10. def addUniqueId(u: UniqueID): Config

    Permalink
  11. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any
  12. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  13. def ensureUniqueId: (UniqueID, Config)

    Permalink

    Allocate a new UniqueID if there is not one present

  14. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  15. def equals(that: Any): Boolean

    Permalink
    Definition Classes
    Config → AnyRef → Any
  16. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  17. def get(key: String): Option[String]

    Permalink
  18. def getArgs: Args

    Permalink
  19. def getCascadingAppJar: Option[Try[Class[_]]]

    Permalink

    Returns None if not set, otherwise reflection is used to create the Class.forName

  20. def getCascadingAppName: Option[String]

    Permalink

    This is a name that if present is passed to flow.setName, which should appear in the job tracker.

  21. def getCascadingSerializationTokens: Map[Int, String]

    Permalink
  22. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any
  23. def getKryo: Option[KryoInstantiator]

    Permalink
  24. def getKryoRegisteredClasses: Set[Class[_]]

    Permalink

    This function gets the set of classes that have been registered to Kryo.

    This function gets the set of classes that have been registered to Kryo. They may or may not be used in this job, but Cascading might want to be made aware that these classes exist

  25. def getNumReducers: Option[Int]

    Permalink

    Get the number of reducers (this is the parameter Hadoop will use)

  26. def getRequireOrderedSerialization: Boolean

    Permalink
  27. def getScaldingExecutionId: Option[String]

    Permalink
  28. def getScaldingVersion: Option[String]

    Permalink
  29. def getSubmittedTimestamp: Option[RichDate]

    Permalink
  30. def getUniqueIds: Set[UniqueID]

    Permalink
  31. def hashCode(): Int

    Permalink
    Definition Classes
    Config → AnyRef → Any
  32. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any
  33. def maybeSetSubmittedTimestamp(date: RichDate = RichDate.now): (Option[RichDate], Config)

    Permalink
  34. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  35. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef
  36. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef
  37. def setArgs(args: Args): Config

    Permalink
  38. def setCascadingAppId(id: String): Config

    Permalink
  39. def setCascadingAppJar(clazz: Class[_]): Config

    Permalink

    Non-fat-jar use cases require this, BUT using it with fat jars can cause problems.

    Non-fat-jar use cases require this, BUT using it with fat jars can cause problems. It is not set by default, but if you have problems you might need to set the Job class here Consider also setting this same class here: setScaldingFlowClass

  40. def setCascadingAppName(name: String): Config

    Permalink
  41. def setDefaultComparator(clazz: Class[_ <: Comparator[_]]): Config

    Permalink
  42. def setHRavenHistoryUserName: Config

    Permalink

    Set username from System.used for querying hRaven.

  43. def setListSpillThreshold(count: Int): Config

    Permalink
  44. def setMapSideAggregationThreshold(count: Int): Config

    Permalink
  45. def setMapSpillThreshold(count: Int): Config

    Permalink
  46. def setNumReducers(n: Int): Config

    Permalink
  47. def setReducerEstimators(clsList: String): Config

    Permalink

    Set the entire list of reducer estimators (overriding the existing list)

  48. def setRequireOrderedSerialization(b: Boolean): Config

    Permalink

    Set this configuration option to require all grouping/cogrouping to use OrderedSerialization

  49. def setScaldingExecutionId(id: String): Config

    Permalink

    Set an ID to be shared across this usage of run for Execution

  50. def setScaldingFlowClass(clazz: Class[_]): Config

    Permalink
  51. def setScaldingVersion: Config

    Permalink
  52. def setSerialization(kryo: Either[(Class[_ <: KryoInstantiator], KryoInstantiator), Class[_ <: KryoInstantiator]], userHadoop: Seq[Class[_ <: Serialization[_]]] = Nil): Config

    Permalink
  53. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef
  54. def toString(): String

    Permalink
    Definition Classes
    AnyRef → Any
  55. def update[R](k: String)(fn: (Option[String]) ⇒ (Option[String], R)): (R, Config)

    Permalink
  56. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  57. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  58. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from Serializable

Inherited from Serializable

Inherited from AnyRef

Inherited from Any

Ungrouped