Class/Object

com.twitter.scalding

JsonLine

Related Docs: object JsonLine | package scalding

Permalink

case class JsonLine(p: String, fields: Fields = Fields.ALL, sinkMode: SinkMode = SinkMode.REPLACE, transformInTest: Boolean = false, failOnEmptyLines: Boolean = true) extends FixedPathSource with TextLineScheme with Product with Serializable

This Source writes out the TupleEntry as a simple JSON object, using the field names as keys and the string representation of the values.

TODO: it would be nice to have a way to add read/write transformations to pipes that doesn't require extending the sources and overriding methods.

failOnEmptyLines

When set to false, it just skips empty lines instead of failing the jobs. Defaults to true for backwards compatibility.

Linear Supertypes
Serializable, Product, Equals, TextLineScheme, SingleMappable[String], Mappable[String], typed.TypedSource[String], TextSourceScheme, FixedPathSource, FileSource, HfsTapProvider, LocalSourceOverride, SchemedSource, Source, Serializable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. JsonLine
  2. Serializable
  3. Product
  4. Equals
  5. TextLineScheme
  6. SingleMappable
  7. Mappable
  8. TypedSource
  9. TextSourceScheme
  10. FixedPathSource
  11. FileSource
  12. HfsTapProvider
  13. LocalSourceOverride
  14. SchemedSource
  15. Source
  16. Serializable
  17. AnyRef
  18. Any
  1. Hide All
  2. Show all
Visibility
  1. Public
  2. All

Instance Constructors

  1. new JsonLine(p: String, fields: Fields = Fields.ALL, sinkMode: SinkMode = SinkMode.REPLACE, transformInTest: Boolean = false, failOnEmptyLines: Boolean = true)

    Permalink

    failOnEmptyLines

    When set to false, it just skips empty lines instead of failing the jobs. Defaults to true for backwards compatibility.

Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any

  3. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  4. def andThen[U](fn: (String) ⇒ U): typed.TypedSource[U]

    Permalink
    Definition Classes
    TypedSource

  5. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any

  6. def checkFlowDefNotNull()(implicit flowDef: FlowDef, mode: Mode): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Source

  7. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  8. def converter[U >: String]: TupleConverter[U]

    Permalink
    Definition Classes
    SingleMappable → TypedSource

  9. def createHdfsReadTap(hdfsMode: Hdfs): Tap[JobConf, _, _]

    Permalink
    Attributes
    protected
    Definition Classes
    FileSource

  10. def createHfsTap(scheme: Scheme[JobConf, RecordReader[_, _], OutputCollector[_, _], _, _], path: String, sinkMode: SinkMode): Hfs

    Permalink
    Definition Classes
    HfsTapProvider

  11. def createLocalTap(sinkMode: SinkMode): Tap[JobConf, _, _]

    Permalink
    Definition Classes
    LocalSourceOverride

  12. def createTap(readOrWrite: AccessMode)(implicit mode: Mode): Tap[_, _, _]

    Permalink
    Definition Classes
    FileSource → Source

  13. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef

  14. def equals(that: Any): Boolean

    Permalink
    Definition Classes
    FixedPathSource → AnyRef → Any

  15. val failOnEmptyLines: Boolean

    Permalink

    When set to false, it just skips empty lines instead of failing the jobs.

    When set to false, it just skips empty lines instead of failing the jobs. Defaults to true for backwards compatibility.

  16. val fields: Fields

    Permalink

  17. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )

  18. final def flatMapTo[U](out: Fields)(mf: (String) ⇒ TraversableOnce[U])(implicit flowDef: FlowDef, mode: Mode, setter: TupleSetter[U]): Pipe

    Permalink
    Definition Classes
    Mappable

  19. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any

  20. def goodHdfsPaths(hdfsMode: Hdfs): Iterable[String]

    Permalink
    Attributes
    protected
    Definition Classes
    FileSource

  21. def hashCode(): Int

    Permalink
    Definition Classes
    FixedPathSource → AnyRef → Any

  22. def hdfsPaths: Iterable[String]

    Permalink
    Definition Classes
    FixedPathSource → FileSource

  23. def hdfsReadPathsAreGood(conf: Configuration): Boolean

    Permalink
    Attributes
    protected
    Definition Classes
    FileSource

  24. def hdfsScheme: Scheme[JobConf, RecordReader[_, _], OutputCollector[_, _], _, _]

    Permalink
    Definition Classes
    TextSourceScheme → SchemedSource

  25. def hdfsWritePath: String

    Permalink
    Definition Classes
    FixedPathSource → FileSource

  26. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any

  27. def localPaths: Iterable[String]

    Permalink
    Definition Classes
    FixedPathSource → LocalSourceOverride

  28. def localScheme: cascading.scheme.local.TextLine

    Permalink
    Definition Classes
    TextSourceScheme → SchemedSource

  29. def localWritePath: String

    Permalink
    Definition Classes
    LocalSourceOverride

  30. final def mapTo[U](out: Fields)(mf: (String) ⇒ U)(implicit flowDef: FlowDef, mode: Mode, setter: TupleSetter[U]): Pipe

    Permalink
    Definition Classes
    Mappable

  31. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef

  32. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef

  33. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef

  34. val p: String

    Permalink

  35. def pathIsGood(p: String, conf: Configuration): Boolean

    Permalink
    Attributes
    protected
    Definition Classes
    FileSource

  36. def read(implicit flowDef: FlowDef, mode: Mode): Pipe

    Permalink
    Definition Classes
    Source

  37. val sinkMode: SinkMode

    Permalink
    Definition Classes
    JsonLine → SchemedSource

  38. def sourceFields: Fields

    Permalink
    Definition Classes
    TextLineScheme → TypedSource

  39. def sourceId: String

    Permalink
    Definition Classes
    Source

  40. def stripTrailing(path: String): String

    Permalink
    Attributes
    protected
    Definition Classes
    FixedPathSource

  41. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef

  42. val textEncoding: String

    Permalink
    Definition Classes
    TextSourceScheme

  43. def toIterator(implicit config: Config, mode: Mode): Iterator[String]

    Permalink
    Definition Classes
    Mappable

  44. def toString(): String

    Permalink
    Definition Classes
    JsonLine → FixedPathSource → AnyRef → Any

  45. def transformForRead(pipe: Pipe): Pipe

    Permalink
    Definition Classes
    JsonLine → Source

  46. def transformForWrite(pipe: Pipe): Pipe

    Permalink
    Definition Classes
    JsonLine → Source

  47. val transformInTest: Boolean

    Permalink
    Definition Classes
    JsonLine → Source

  48. def validateTaps(mode: Mode): Unit

    Permalink
    Definition Classes
    FileSource → Source

  49. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  50. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  51. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  52. def writeFrom(pipe: Pipe)(implicit flowDef: FlowDef, mode: Mode): Pipe

    Permalink
    Definition Classes
    Source

Deprecated Value Members

  1. def readAtSubmitter[T](implicit mode: Mode, conv: TupleConverter[T]): Stream[T]

    Permalink
    Definition Classes
    Source
    Annotations
    @deprecated
    Deprecated

    (Since version 0.9.0) replace with Mappable.toIterator

Inherited from Serializable

Inherited from Product

Inherited from Equals

Inherited from TextLineScheme

Inherited from SingleMappable[String]

Inherited from Mappable[String]

Inherited from typed.TypedSource[String]

Inherited from TextSourceScheme

Inherited from FixedPathSource

Inherited from FileSource

Inherited from HfsTapProvider

Inherited from LocalSourceOverride

Inherited from SchemedSource

Inherited from Source

Inherited from Serializable

Inherited from AnyRef

Inherited from Any

Ungrouped