DeequActionImplicits

Value Members

1. final def !=(arg0: Any): Boolean

   

   Definition Classes

   AnyRef → Any

2. final def ##(): Int

   

   Definition Classes

   AnyRef → Any

3. final def ==(arg0: Any): Boolean

   

   Definition Classes

   AnyRef → Any

4. def addDeequCheck(label: String, check: Check, checks: Check*)(alertHandler: DataQualityAlertHandler, alertHandlers: DataQualityAlertHandler*): SparkDataFlow

   

   Adds Deequ Checks for a label

   Adds Deequ Checks for a label

   label

   the label to perform the validation for

   check

   first Deequ Check to perform

   checks

   additional Deequ checks to perform

   alertHandler

   the alert handler to use for handling alerts for this check

   alertHandlers

   additional alert handlers to use

5. def addDeequValidation(label: String, checks: (VerificationRunBuilder) ⇒ VerificationRunBuilder, alertHandler: DataQualityAlertHandler, alertHandlers: DataQualityAlertHandler*): SparkDataFlow

   

   Add Deequ validation for a given label (https://github.com/awslabs/deequ)

   Add Deequ validation for a given label (https://github.com/awslabs/deequ)

   label

   the label to perform the validation for

   checks

   the Deequ validation to perform e.g. _.addCheck( Check(CheckLevel.Error, "unit testing my data") .hasSize(_ == 5) // we expect 5 rows .isComplete("id") // should never be NULL )

   alertHandler

   the alert handler to use for handling alerts for this check

   alertHandlers

   additional alert handlers to use

6. def addDeequValidationWithMetrics(label: String, checksWithMetrics: (VerificationRunBuilderWithRepository) ⇒ VerificationRunBuilderWithRepository, alertHandler: DataQualityAlertHandler, alertHandlers: DataQualityAlertHandler*): SparkDataFlow

   

   Adds Deequ validation which uses a metrics repository (see Deequ docs on metrics repositories for more information) N.B in order for this to work, you must set a metrics repository using setDeequMetricsRepository or setDeequStorageLayerMetricsRepository

   Adds Deequ validation which uses a metrics repository (see Deequ docs on metrics repositories for more information) N.B in order for this to work, you must set a metrics repository using setDeequMetricsRepository or setDeequStorageLayerMetricsRepository

   label

   the label to perform the validation for

   checksWithMetrics

   the Deequ validation to perform e.g. _.addAnomalyCheck(RateOfChangeStrategy(maxRateDecrease = Some(0.2)), Completeness("col1")))

   alertHandler

   the alert handler to use for handling alerts for this check

   alertHandlers

   additional alert handlers to use

7. final def asInstanceOf[T0]: T0

   

   Definition Classes

   Any

8. def clone(): AnyRef

   

   Attributes

   protected[java.lang]

   Definition Classes

   AnyRef

   Annotations

   @throws( ... )

9. final def eq(arg0: AnyRef): Boolean

   

   Definition Classes

   AnyRef

10. def equals(arg0: Any): Boolean

    

    Definition Classes

    AnyRef → Any

11. def finalize(): Unit

    

    Attributes

    protected[java.lang]

    Definition Classes

    AnyRef

    Annotations

    @throws( classOf[java.lang.Throwable] )

12. final def getClass(): Class[_]

    

    Definition Classes

    AnyRef → Any

13. def hashCode(): Int

    

    Definition Classes

    AnyRef → Any

14. final def isInstanceOf[T0]: Boolean

    

    Definition Classes

    Any

15. final def ne(arg0: AnyRef): Boolean

    

    Definition Classes

    AnyRef

16. final def notify(): Unit

    

    Definition Classes

    AnyRef

17. final def notifyAll(): Unit

    

    Definition Classes

    AnyRef

18. def setDeequMetricsRepository(builder: DeequMetricsRepositoryBuilder, appendDateTime: ZonedDateTime = ZonedDateTime.now()): SparkDataFlow

    

    Sets the Deequ metrics repository to use for for Deequ checks using metrics, specified using addDeequValidationWithMetrics

    Sets the Deequ metrics repository to use for for Deequ checks using metrics, specified using addDeequValidationWithMetrics

    builder

    Deequ metrics repository builder

    appendDateTime

    the date time to be used as a key in the metrics repository (defaults to now)

19. def setDeequStorageLayerMetricsRepository(storageBasePath: String, appendDateTime: ZonedDateTime = ZonedDateTime.now()): SparkDataFlow

    

    Sets the Deequ metrics repository to use for for Deequ checks using metrics, specified using addDeequValidationWithMetrics The metrics repository will be a StorageLayerMetricsRepository

    Sets the Deequ metrics repository to use for for Deequ checks using metrics, specified using addDeequValidationWithMetrics The metrics repository will be a StorageLayerMetricsRepository

    storageBasePath

    the base path to use for the StorageLayerMetricsRepository

    appendDateTime

    the date time to be used as a key in the metrics repository (defaults to now)

20. final def synchronized[T0](arg0: ⇒ T0): T0

    

    Definition Classes

    AnyRef

21. def toString(): String

    

    Definition Classes

    AnyRef → Any

22. final def wait(): Unit

    

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

23. final def wait(arg0: Long, arg1: Int): Unit

    

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

24. final def wait(arg0: Long): Unit

    

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )