PartitionedTextLine

Scalding source to read or write partitioned text.

For writing it expects a pair of (P, String), where P is the data used for partitioning and String is the output to write out. Below is an example.

val data = List(
  (("a", "x"), "line1"),
  (("a", "y"), "line2"),
  (("b", "z"), "line3")
)
IterablePipe(data, flowDef, mode)
  .write(PartitionTextLine[(String, String)](args("out"), "col1=%s/col2=%s"))

For reading it produces a pair (P, (Long, String)) where P is the partition data, Long is the offset into the file and String is a line from the file. Below is an example.

val in: TypedPipe[((String, String), (Long, String))] = PartitionTextLine[(String, String)](args("in"), "col1=%s/col2=%s")

path: Base path of the partitioned directory
template: Template for the partitioned path
encoding: Text encoding of the file content

Linear Supertypes

Serializable, Product, Equals, HfsTapProvider, Mappable[(P, (Long, String))], TypedSource[(P, (Long, String))], TypedSink[(P, String)], SchemedSource, Source, Serializable, AnyRef, Any

Instance Constructors

new PartitionedTextLine(path: String, template: String, encoding: String = ...)(implicit valueSetter: TupleSetter[String], valueConverter: TupleConverter[(Long, String)], partitionSetter: TupleSetter[P], partitionConverter: TupleConverter[P])

path
Base path of the partitioned directory
template
Template for the partitioned path
encoding
Text encoding of the file content

Value Members

final def !=(arg0: AnyRef): Boolean

Definition Classes
AnyRef
final def !=(arg0: Any): Boolean

Definition Classes
Any
final def ##(): Int

Definition Classes
AnyRef → Any
final def ==(arg0: AnyRef): Boolean

Definition Classes
AnyRef
final def ==(arg0: Any): Boolean

Definition Classes
Any
def andThen[U](fn: ((P, (Long, String))) ⇒ U): TypedSource[U]

Transform this TypedSource into another by mapping after.
Transform this TypedSource into another by mapping after. We don't call this map because of conflicts with Mappable, unfortunately

Definition Classes
TypedSource
final def asInstanceOf[T0]: T0

Definition Classes
Any
def checkFlowDefNotNull()(implicit flowDef: FlowDef, mode: Mode): Unit

Attributes
protected
Definition Classes
Source
def clone(): AnyRef

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( ... )
def contraMap[U](fn: (U) ⇒ (P, String)): TypedSink[U]

Transform this sink into another type by applying a function first
Transform this sink into another type by applying a function first

Definition Classes
TypedSink
def converter[U >: (P, (Long, String))]: TupleConverter[U]

Combine both the partition and value converter to extract the data from a flat cascading tuple into a pair of P and (offset, line).
Combine both the partition and value converter to extract the data from a flat cascading tuple into a pair of P and (offset, line).

Definition Classes
PartitionedTextLine → TypedSource
def createHfsTap(scheme: Scheme[JobConf, RecordReader[_, _], OutputCollector[_, _], _, _], path: String, sinkMode: SinkMode): Hfs

Definition Classes
HfsTapProvider
def createTap(readOrWrite: AccessMode)(implicit mode: Mode): Tap[_, _, _]

Creates the taps for local and hdfs mode.
Creates the taps for local and hdfs mode.

Definition Classes
PartitionedTextLine → Source
val encoding: String

Text encoding of the file content
final def eq(arg0: AnyRef): Boolean

Definition Classes
AnyRef
def finalize(): Unit

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( classOf[java.lang.Throwable] )
final def flatMapTo[U](out: Fields)(mf: ((P, (Long, String))) ⇒ TraversableOnce[U])(implicit flowDef: FlowDef, mode: Mode, setter: TupleSetter[U]): Pipe

If you want to filter, you should use this and output a 0 or 1 length Iterable.
If you want to filter, you should use this and output a 0 or 1 length Iterable. Filter does not change column names, and we generally expect to change columns here

Definition Classes
Mappable
final def getClass(): Class[_]

Definition Classes
AnyRef → Any
def hdfsScheme: Scheme[JobConf, RecordReader[_, _], OutputCollector[_, _], _, _]

The scheme to use if the source is on hdfs.
The scheme to use if the source is on hdfs.

Definition Classes
PartitionedTextLine → SchemedSource
final def isInstanceOf[T0]: Boolean

Definition Classes
Any
def localScheme: Scheme[Properties, InputStream, OutputStream, _, _]

The scheme to use if the source is local.
The scheme to use if the source is local.

Definition Classes
PartitionedTextLine → SchemedSource
final def mapTo[U](out: Fields)(mf: ((P, (Long, String))) ⇒ U)(implicit flowDef: FlowDef, mode: Mode, setter: TupleSetter[U]): Pipe

Definition Classes
Mappable
final def ne(arg0: AnyRef): Boolean

Definition Classes
AnyRef
final def notify(): Unit

Definition Classes
AnyRef
final def notifyAll(): Unit

Definition Classes
AnyRef
implicit val partitionConverter: TupleConverter[P]
val partitionFields: Fields
implicit val partitionSetter: TupleSetter[P]
val path: String

Base path of the partitioned directory
def read(implicit flowDef: FlowDef, mode: Mode): Pipe

Definition Classes
Source
def setter[U <: (P, String)]: TupleSetter[U]

Flatten a pair of P and line into a cascading tuple.
Flatten a pair of P and line into a cascading tuple.

Definition Classes
PartitionedTextLine → TypedSink
def sinkFields: Fields

Definition Classes
PartitionedTextLine → TypedSink
val sinkMode: SinkMode

Definition Classes
SchemedSource
def sourceFields: Fields

Definition Classes
TypedSource
def sourceId: String

This is a name the refers to this exact instance of the source (put another way, if s1.
This is a name the refers to this exact instance of the source (put another way, if s1.sourceId == s2.sourceId, the job should work the same if one is replaced with the other

Definition Classes
Source
final def synchronized[T0](arg0: ⇒ T0): T0

Definition Classes
AnyRef
val template: String

Template for the partitioned path
def toIterator(implicit config: Config, mode: Mode): Iterator[(P, (Long, String))]

Allows you to read a Tap on the submit node NOT FOR USE IN THE MAPPERS OR REDUCERS.
Allows you to read a Tap on the submit node NOT FOR USE IN THE MAPPERS OR REDUCERS. Typical use might be to read in Job.next to determine if another job is needed

Definition Classes
Mappable
def transformForRead(pipe: Pipe): Pipe

Attributes
protected
Definition Classes
Source
def transformForWrite(pipe: Pipe): Pipe

Attributes
protected
Definition Classes
Source
def transformInTest: Boolean

The mock passed in to scalding.
The mock passed in to scalding.JobTest may be considered as a mock of the Tap or the Source. By default, as of 0.9.0, it is considered as a Mock of the Source. If you set this to true, the mock in TestMode will be considered to be a mock of the Tap (which must be transformed) and not the Source.

Definition Classes
Source
def validateTaps(mode: Mode): Unit

Definition Classes
Source
implicit val valueConverter: TupleConverter[(Long, String)]
implicit val valueSetter: TupleSetter[String]
final def wait(): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long, arg1: Int): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
def writeFrom(pipe: Pipe)(implicit flowDef: FlowDef, mode: Mode): Pipe

write the pipe but return the input so it can be chained into the next operation
write the pipe but return the input so it can be chained into the next operation

Definition Classes
Source

Deprecated Value Members

def readAtSubmitter[T](implicit mode: Mode, conv: TupleConverter[T]): Stream[T]

Definition Classes
Source
Annotations
@deprecated
Deprecated
(Since version 0.9.0) replace with Mappable.toIterator

Instance Constructors

new PartitionedTextLine(path: String, template: String, encoding: String = ...)(implicit valueSetter: TupleSetter[String], valueConverter: TupleConverter[(Long, String)], partitionSetter: TupleSetter[P], partitionConverter: TupleConverter[P])

Value Members

final def !=(arg0: AnyRef): Boolean

final def !=(arg0: Any): Boolean

final def ##(): Int

final def ==(arg0: AnyRef): Boolean

final def ==(arg0: Any): Boolean

def andThen[U](fn: ((P, (Long, String))) ⇒ U): TypedSource[U]

final def asInstanceOf[T0]: T0

def checkFlowDefNotNull()(implicit flowDef: FlowDef, mode: Mode): Unit

def clone(): AnyRef

def contraMap[U](fn: (U) ⇒ (P, String)): TypedSink[U]

def converter[U >: (P, (Long, String))]: TupleConverter[U]

def createHfsTap(scheme: Scheme[JobConf, RecordReader[_, _], OutputCollector[_, _], _, _], path: String, sinkMode: SinkMode): Hfs

def createTap(readOrWrite: AccessMode)(implicit mode: Mode): Tap[_, _, _]

val encoding: String

final def eq(arg0: AnyRef): Boolean

def finalize(): Unit

final def flatMapTo[U](out: Fields)(mf: ((P, (Long, String))) ⇒ TraversableOnce[U])(implicit flowDef: FlowDef, mode: Mode, setter: TupleSetter[U]): Pipe

final def getClass(): Class[_]

def hdfsScheme: Scheme[JobConf, RecordReader[_, _], OutputCollector[_, _], _, _]

final def isInstanceOf[T0]: Boolean

def localScheme: Scheme[Properties, InputStream, OutputStream, _, _]

final def mapTo[U](out: Fields)(mf: ((P, (Long, String))) ⇒ U)(implicit flowDef: FlowDef, mode: Mode, setter: TupleSetter[U]): Pipe

final def ne(arg0: AnyRef): Boolean

final def notify(): Unit

final def notifyAll(): Unit

implicit val partitionConverter: TupleConverter[P]

val partitionFields: Fields

implicit val partitionSetter: TupleSetter[P]

val path: String

def read(implicit flowDef: FlowDef, mode: Mode): Pipe

def setter[U <: (P, String)]: TupleSetter[U]

def sinkFields: Fields

val sinkMode: SinkMode

def sourceFields: Fields

def sourceId: String

final def synchronized[T0](arg0: ⇒ T0): T0

val template: String

def toIterator(implicit config: Config, mode: Mode): Iterator[(P, (Long, String))]

def transformForRead(pipe: Pipe): Pipe

def transformForWrite(pipe: Pipe): Pipe

def transformInTest: Boolean

def validateTaps(mode: Mode): Unit

implicit val valueConverter: TupleConverter[(Long, String)]

implicit val valueSetter: TupleSetter[String]

final def wait(): Unit

final def wait(arg0: Long, arg1: Int): Unit

final def wait(arg0: Long): Unit

def writeFrom(pipe: Pipe)(implicit flowDef: FlowDef, mode: Mode): Pipe

Deprecated Value Members

def readAtSubmitter[T](implicit mode: Mode, conv: TupleConverter[T]): Stream[T]

Inherited from Serializable

Inherited from Product

Inherited from Equals

Inherited from HfsTapProvider

Inherited from Mappable[(P, (Long, String))]

Inherited from TypedSource[(P, (Long, String))]

Inherited from TypedSink[(P, String)]

Inherited from SchemedSource

Inherited from Source

Inherited from Serializable

Inherited from AnyRef

Inherited from Any

Ungrouped