Class/Object

io.smartdatalake.workflow.action.sparktransformer

DfTransformerWrapperDfsTransformer

Related Docs: object DfTransformerWrapperDfsTransformer | package sparktransformer

Permalink

case class DfTransformerWrapperDfsTransformer(transformer: ParsableDfTransformer, subFeedsToApply: Seq[String]) extends ParsableDfsTransformer with Product with Serializable

A Transformer to use single DataFrame Transformers as multiple DataFrame Transformers. This works by selecting the SubFeeds (DataFrames) the single DataFrame Transformer should be applied to. All other SubFeeds will be passed through without transformation.

transformer

Configuration for a DfTransformer to be applied

subFeedsToApply

Names of SubFeeds the transformation should be applied to.

Linear Supertypes
Serializable, Serializable, Product, Equals, ParsableDfsTransformer, ParsableFromConfig[ParsableDfsTransformer], DfsTransformer, PartitionValueTransformer, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. DfTransformerWrapperDfsTransformer
  2. Serializable
  3. Serializable
  4. Product
  5. Equals
  6. ParsableDfsTransformer
  7. ParsableFromConfig
  8. DfsTransformer
  9. PartitionValueTransformer
  10. AnyRef
  11. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new DfTransformerWrapperDfsTransformer(transformer: ParsableDfTransformer, subFeedsToApply: Seq[String])

    Permalink

    transformer

    Configuration for a DfTransformer to be applied

    subFeedsToApply

    Names of SubFeeds the transformation should be applied to.

Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any

  3. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  4. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any

  5. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  6. def description: Option[String]

    Permalink
    Definition Classes
    DfTransformerWrapperDfsTransformer → DfsTransformer

  7. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef

  8. def factory: FromConfigFactory[ParsableDfsTransformer]

    Permalink

    Returns the factory that can parse this type (that is, type CO).

    Returns the factory that can parse this type (that is, type CO).

    Typically, implementations of this method should return the companion object of the implementing class. The companion object in turn should implement FromConfigFactory.

    returns

    the factory (object) for this class.

    Definition Classes
    DfTransformerWrapperDfsTransformer → ParsableFromConfig

  9. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )

  10. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any

  11. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any

  12. def name: String

    Permalink
    Definition Classes
    DfTransformerWrapperDfsTransformer → DfsTransformer

  13. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef

  14. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef

  15. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef

  16. def prepare(actionId: ActionId)(implicit session: SparkSession, context: ActionPipelineContext): Unit

    Permalink

    Optional function to implement validations in prepare phase.

    Optional function to implement validations in prepare phase.

    Definition Classes
    DfsTransformer

  17. val subFeedsToApply: Seq[String]

    Permalink

    Names of SubFeeds the transformation should be applied to.

  18. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef

  19. def transform(actionId: ActionId, partitionValues: Seq[PartitionValues], dfs: Map[String, DataFrame])(implicit session: SparkSession, context: ActionPipelineContext): Map[String, DataFrame]

    Permalink

    Function to be implemented to define the transformation between many inputs and many outputs (n:m)

    Function to be implemented to define the transformation between many inputs and many outputs (n:m)

    actionId

    id of the action which executes this transformation. This is mainly used to prefix error messages.

    partitionValues

    partition values to transform

    dfs

    Map of (dataObjectId, DataFrame) tuples available as input

    returns

    Map of transformed (dataObjectId, DataFrame) tuples

    Definition Classes
    DfTransformerWrapperDfsTransformer → DfsTransformer

  20. def transformPartitionValues(actionId: ActionId, partitionValues: Seq[PartitionValues])(implicit session: SparkSession, context: ActionPipelineContext): Option[Map[PartitionValues, PartitionValues]]

    Permalink

    Optional function to define the transformation of input to output partition values.

    Optional function to define the transformation of input to output partition values. For example this enables to implement aggregations where multiple input partitions are combined into one output partition. Note that the default value is input = output partition values, which should be correct for most use cases.

    actionId

    id of the action which executes this transformation. This is mainly used to prefix error messages.

    partitionValues

    partition values to transform

    returns

    Map of input to output partition values. This allows to map partition values forward and backward, which is needed in execution modes. Return None if mapping is 1:1.

    Definition Classes
    DfTransformerWrapperDfsTransformer → PartitionValueTransformer

  21. val transformer: ParsableDfTransformer

    Permalink

    Configuration for a DfTransformer to be applied

  22. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  23. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  24. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from Serializable

Inherited from Serializable

Inherited from Product

Inherited from Equals

Inherited from ParsableDfsTransformer

Inherited from ParsableFromConfig[ParsableDfsTransformer]

Inherited from DfsTransformer

Inherited from PartitionValueTransformer

Inherited from AnyRef

Inherited from Any

Ungrouped