com.twitter.scalding

PartitionSource

abstract class PartitionSource extends SchemedSource with HfsTapProvider

This is a base class for partition-based output sources

Linear Supertypes
HfsTapProvider, SchemedSource, Source, Serializable, AnyRef, Any
Known Subclasses
PartitionedSequenceFile, PartitionedTsv
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. PartitionSource
  2. HfsTapProvider
  3. SchemedSource
  4. Source
  5. Serializable
  6. AnyRef
  7. Any
  1. Hide All
  2. Show all
Learn more about member selection
Visibility
  1. Public
  2. All

Instance Constructors

  1. new PartitionSource(openWritesThreshold: Option[Int] = scala.None)

Abstract Value Members

  1. abstract def basePath: String

Concrete Value Members

  1. final def !=(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  2. final def !=(arg0: Any): Boolean

    Definition Classes
    Any

  3. final def ##(): Int

    Definition Classes
    AnyRef → Any

  4. final def ==(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  5. final def ==(arg0: Any): Boolean

    Definition Classes
    Any

  6. final def asInstanceOf[T0]: T0

    Definition Classes
    Any

  7. def checkFlowDefNotNull()(implicit flowDef: FlowDef, mode: Mode): Unit

    Attributes
    protected
    Definition Classes
    Source

  8. def clone(): AnyRef

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  9. def createHfsTap(scheme: Scheme[JobConf, RecordReader[_, _], OutputCollector[_, _], _, _], path: String, sinkMode: SinkMode): Hfs

    Definition Classes
    HfsTapProvider

  10. def createTap(readOrWrite: AccessMode)(implicit mode: Mode): Tap[_, _, _]

    Creates the partition tap.

    Creates the partition tap.

    readOrWrite

    Describes if this source is being read from or written to.

    mode

    The mode of the job. (implicit)

    Definition Classes
    PartitionSource → Source

  11. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  12. def equals(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any

  13. def finalize(): Unit

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )

  14. final def getClass(): Class[_]

    Definition Classes
    AnyRef → Any

  15. def hashCode(): Int

    Definition Classes
    AnyRef → Any

  16. def hdfsScheme: Scheme[JobConf, RecordReader[_, _], OutputCollector[_, _], _, _]

    The scheme to use if the source is on hdfs.

    The scheme to use if the source is on hdfs.

    Definition Classes
    SchemedSource

  17. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any

  18. def localScheme: Scheme[Properties, InputStream, OutputStream, _, _]

    The scheme to use if the source is local.

    The scheme to use if the source is local.

    Definition Classes
    SchemedSource

  19. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  20. final def notify(): Unit

    Definition Classes
    AnyRef

  21. final def notifyAll(): Unit

    Definition Classes
    AnyRef

  22. val openWritesThreshold: Option[Int]

  23. def partition: Partition

  24. def read(implicit flowDef: FlowDef, mode: Mode): Pipe

    Definition Classes
    Source

  25. val sinkMode: SinkMode

    Definition Classes
    SchemedSource

  26. def sourceId: String

    This is a name the refers to this exact instance of the source (put another way, if s1.

    This is a name the refers to this exact instance of the source (put another way, if s1.sourceId == s2.sourceId, the job should work the same if one is replaced with the other

    Definition Classes
    Source

  27. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef

  28. def toString(): String

    Definition Classes
    AnyRef → Any

  29. def transformForRead(pipe: Pipe): Pipe

    Attributes
    protected
    Definition Classes
    Source

  30. def transformForWrite(pipe: Pipe): Pipe

    Attributes
    protected
    Definition Classes
    Source

  31. def transformInTest: Boolean

    The mock passed in to scalding.

    The mock passed in to scalding.JobTest may be considered as a mock of the Tap or the Source. By default, as of 0.9.0, it is considered as a Mock of the Source. If you set this to true, the mock in TestMode will be considered to be a mock of the Tap (which must be transformed) and not the Source.

    Definition Classes
    Source

  32. def validateTaps(mode: Mode): Unit

    Validates the taps, makes sure there are no nulls in the path.

    Validates the taps, makes sure there are no nulls in the path.

    mode

    The mode of the job.

    Definition Classes
    PartitionSource → Source

  33. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  34. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  35. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  36. def writeFrom(pipe: Pipe)(implicit flowDef: FlowDef, mode: Mode): Pipe

    write the pipe but return the input so it can be chained into the next operation

    write the pipe but return the input so it can be chained into the next operation

    Definition Classes
    Source

Deprecated Value Members

  1. def readAtSubmitter[T](implicit mode: Mode, conv: TupleConverter[T]): Stream[T]

    Definition Classes
    Source
    Annotations
    @deprecated
    Deprecated

    (Since version 0.9.0) replace with Mappable.toIterator

Inherited from HfsTapProvider

Inherited from SchemedSource

Inherited from Source

Inherited from Serializable

Inherited from AnyRef

Inherited from Any

Ungrouped