com.twitter.scalding

TemplatedSequenceFile

case class TemplatedSequenceFile(basePath: String, template: String, sequenceFields: Fields = cascading.tuple.Fields.ALL, pathFields: Fields = cascading.tuple.Fields.ALL, sinkMode: SinkMode = cascading.tap.SinkMode.REPLACE) extends TemplateSource with SequenceFileScheme with Product with Serializable

An implementation of SequenceFile output, split over a template tap.

basePath

The root path for the output.

template

The java formatter style string to use as the template. e.g. %s/%s.

sequenceFields

The set of fields to use for the sequence file.

pathFields

The set of fields to apply to the path.

sinkMode

How to handle conflicts with existing output.

Linear Supertypes
Serializable, Product, Equals, SequenceFileScheme, TemplateSource, HfsTapProvider, SchemedSource, Source, Serializable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. TemplatedSequenceFile
  2. Serializable
  3. Product
  4. Equals
  5. SequenceFileScheme
  6. TemplateSource
  7. HfsTapProvider
  8. SchemedSource
  9. Source
  10. Serializable
  11. AnyRef
  12. Any
  1. Hide All
  2. Show all
Learn more about member selection
Visibility
  1. Public
  2. All

Instance Constructors

  1. new TemplatedSequenceFile(basePath: String, template: String, sequenceFields: Fields = cascading.tuple.Fields.ALL, pathFields: Fields = cascading.tuple.Fields.ALL, sinkMode: SinkMode = cascading.tap.SinkMode.REPLACE)

    basePath

    The root path for the output.

    template

    The java formatter style string to use as the template. e.g. %s/%s.

    sequenceFields

    The set of fields to use for the sequence file.

    pathFields

    The set of fields to apply to the path.

    sinkMode

    How to handle conflicts with existing output.

Value Members

  1. final def !=(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  2. final def !=(arg0: Any): Boolean

    Definition Classes
    Any

  3. final def ##(): Int

    Definition Classes
    AnyRef → Any

  4. final def ==(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  5. final def ==(arg0: Any): Boolean

    Definition Classes
    Any

  6. final def asInstanceOf[T0]: T0

    Definition Classes
    Any

  7. val basePath: String

    The root path for the output.

    The root path for the output.

    Definition Classes
    TemplatedSequenceFile → TemplateSource

  8. def checkFlowDefNotNull()(implicit flowDef: FlowDef, mode: Mode): Unit

    Attributes
    protected
    Definition Classes
    Source

  9. def clone(): AnyRef

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  10. def createHfsTap(scheme: Scheme[JobConf, RecordReader[_, _], OutputCollector[_, _], _, _], path: String, sinkMode: SinkMode): Hfs

    Definition Classes
    HfsTapProvider

  11. def createTap(readOrWrite: AccessMode)(implicit mode: Mode): Tap[_, _, _]

    Creates the template tap.

    Creates the template tap.

    readOrWrite

    Describes if this source is being read from or written to.

    mode

    The mode of the job. (implicit)

    Definition Classes
    TemplateSource → Source

  12. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  13. val fields: Fields

    Definition Classes
    TemplatedSequenceFile → SequenceFileScheme

  14. def finalize(): Unit

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )

  15. final def getClass(): Class[_]

    Definition Classes
    AnyRef → Any

  16. def hdfsScheme: Scheme[JobConf, RecordReader[_, _], OutputCollector[_, _], _, _]

    The scheme to use if the source is on hdfs.

    The scheme to use if the source is on hdfs.

    Definition Classes
    SequenceFileScheme → SchemedSource

  17. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any

  18. def localScheme: Scheme[Properties, InputStream, OutputStream, _, _]

    The scheme to use if the source is local.

    The scheme to use if the source is local.

    Definition Classes
    SchemedSource

  19. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  20. final def notify(): Unit

    Definition Classes
    AnyRef

  21. final def notifyAll(): Unit

    Definition Classes
    AnyRef

  22. val pathFields: Fields

    The set of fields to apply to the path.

    The set of fields to apply to the path.

    Definition Classes
    TemplatedSequenceFile → TemplateSource

  23. def read(implicit flowDef: FlowDef, mode: Mode): Pipe

    Definition Classes
    Source

  24. val sequenceFields: Fields

    The set of fields to use for the sequence file.

  25. val sinkMode: SinkMode

    How to handle conflicts with existing output.

    How to handle conflicts with existing output.

    Definition Classes
    TemplatedSequenceFile → SchemedSource

  26. def sourceId: String

    This is a name the refers to this exact instance of the source (put another way, if s1.

    This is a name the refers to this exact instance of the source (put another way, if s1.sourceId == s2.sourceId, the job should work the same if one is replaced with the other

    Definition Classes
    Source

  27. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef

  28. val template: String

    The java formatter style string to use as the template.

    The java formatter style string to use as the template. e.g. %s/%s.

    Definition Classes
    TemplatedSequenceFile → TemplateSource

  29. def transformForRead(pipe: Pipe): Pipe

    Attributes
    protected
    Definition Classes
    Source

  30. def transformForWrite(pipe: Pipe): Pipe

    Attributes
    protected
    Definition Classes
    Source

  31. def transformInTest: Boolean

    The mock passed in to scalding.

    The mock passed in to scalding.JobTest may be considered as a mock of the Tap or the Source. By default, as of 0.9.0, it is considered as a Mock of the Source. If you set this to true, the mock in TestMode will be considered to be a mock of the Tap (which must be transformed) and not the Source.

    Definition Classes
    Source

  32. def validateTaps(mode: Mode): Unit

    Validates the taps, makes sure there are no nulls as the path or template.

    Validates the taps, makes sure there are no nulls as the path or template.

    mode

    The mode of the job.

    Definition Classes
    TemplateSource → Source

  33. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  34. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  35. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  36. def writeFrom(pipe: Pipe)(implicit flowDef: FlowDef, mode: Mode): Pipe

    write the pipe but return the input so it can be chained into the next operation

    write the pipe but return the input so it can be chained into the next operation

    Definition Classes
    Source

Deprecated Value Members

  1. def readAtSubmitter[T](implicit mode: Mode, conv: TupleConverter[T]): Stream[T]

    Definition Classes
    Source
    Annotations
    @deprecated
    Deprecated

    (Since version 0.9.0) replace with Mappable.toIterator

Inherited from Serializable

Inherited from Product

Inherited from Equals

Inherited from SequenceFileScheme

Inherited from TemplateSource

Inherited from HfsTapProvider

Inherited from SchemedSource

Inherited from Source

Inherited from Serializable

Inherited from AnyRef

Inherited from Any

Ungrouped