Packages

t

org.apache.spark.sql.execution.datasources

WriteJobStatsTracker

trait WriteJobStatsTracker extends Serializable

A class implementing this trait is basically a collection of parameters that are necessary for instantiating a (derived type of) WriteTaskStatsTracker on all executors and then process the statistics produced by them (e.g. save them to memory/disk, issue warnings, etc). It is therefore important that such an objects is Serializable, as it will be sent from the driver to all executors.

Linear Supertypes
Serializable, Serializable, AnyRef, Any
Known Subclasses
BasicWriteJobStatsTracker
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. WriteJobStatsTracker
  2. Serializable
  3. Serializable
  4. AnyRef
  5. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Abstract Value Members

  1. abstract def newTaskInstance(): WriteTaskStatsTracker

    Instantiates a WriteTaskStatsTracker, based on (non-transient) members of this class.

    Instantiates a WriteTaskStatsTracker, based on (non-transient) members of this class. To be called by executors.

    returns

    A WriteTaskStatsTracker instance to be used for computing stats during a write task

  2. abstract def processStats(stats: Seq[WriteTaskStats], jobCommitTime: Long): Unit

    Process the given collection of stats computed during this job.

    Process the given collection of stats computed during this job. E.g. aggregate them, write them to memory / disk, issue warnings, whatever.

    stats

    One WriteTaskStats object from each successful write task.

    jobCommitTime

    Time of committing the job.

    Note

    The type of @param stats is too generic. These classes should probably be parametrized: WriteTaskStatsTracker[S <: WriteTaskStats] WriteJobStatsTracker[S <: WriteTaskStats, T <: WriteTaskStatsTracker[S]] and this would then be: def processStats(stats: Seq[S]): Unit but then we wouldn't be able to have a Seq[WriteJobStatsTracker] due to type co-/contra-variance considerations. Instead, you may feel free to just cast stats to the expected derived type when implementing this method in a derived class. The framework will make sure to call this with the right arguments.

Concrete Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  6. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  7. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  8. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  9. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  10. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  11. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  12. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  13. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  14. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  15. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  16. def toString(): String
    Definition Classes
    AnyRef → Any
  17. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  18. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  19. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()

Inherited from Serializable

Inherited from Serializable

Inherited from AnyRef

Inherited from Any

Ungrouped