Trait

com.holdenkarau.spark.testing

ScalaDataFrameSuiteBase

Related Doc: package testing

Permalink

trait ScalaDataFrameSuiteBase extends AnyFunSuite with DataFrameSuiteBase

Base trait for testing Spark DataFrames in Scala.

Linear Supertypes
DataFrameSuiteBase, DataFrameSuiteBaseLike, SharedSparkContext, SparkContextProvider, BeforeAndAfterAll, SuiteMixin, TestSuite, TestSuiteLike, AnyFunSuite, AnyFunSuiteLike, Documenting, Alerting, Notifying, Informing, TestRegistration, org.scalatest.TestSuite, Suite, Serializable, Serializable, Assertions, TripleEquals, TripleEqualsSupport, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. ScalaDataFrameSuiteBase
  2. DataFrameSuiteBase
  3. DataFrameSuiteBaseLike
  4. SharedSparkContext
  5. SparkContextProvider
  6. BeforeAndAfterAll
  7. SuiteMixin
  8. TestSuite
  9. TestSuiteLike
  10. AnyFunSuite
  11. AnyFunSuiteLike
  12. Documenting
  13. Alerting
  14. Notifying
  15. Informing
  16. TestRegistration
  17. TestSuite
  18. Suite
  19. Serializable
  20. Serializable
  21. Assertions
  22. TripleEquals
  23. TripleEqualsSupport
  24. AnyRef
  25. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Type Members

  1. class CheckingEqualizer[L] extends AnyRef

    Permalink
    Definition Classes
    TripleEqualsSupport
  2. class Equalizer[L] extends AnyRef

    Permalink
    Definition Classes
    TripleEqualsSupport
  3. trait NoArgTest extends () ⇒ Outcome with TestData

    Permalink
    Attributes
    protected
    Definition Classes
    TestSuite

Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  2. def !==[T](right: Spread[T]): TripleEqualsInvocationOnSpread[T]

    Permalink
    Definition Classes
    TripleEqualsSupport
  3. def !==(right: Null): TripleEqualsInvocation[Null]

    Permalink
    Definition Classes
    TripleEqualsSupport
  4. def !==[T](right: T): TripleEqualsInvocation[T]

    Permalink
    Definition Classes
    TripleEqualsSupport
  5. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  6. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  7. def ===[T](right: Spread[T]): TripleEqualsInvocationOnSpread[T]

    Permalink
    Definition Classes
    TripleEqualsSupport
  8. def ===(right: Null): TripleEqualsInvocation[Null]

    Permalink
    Definition Classes
    TripleEqualsSupport
  9. def ===[T](right: T): TripleEqualsInvocation[T]

    Permalink
    Definition Classes
    TripleEqualsSupport
  10. def afterAll(): Unit

    Permalink
    Definition Classes
    DataFrameSuiteBaseSharedSparkContext → BeforeAndAfterAll
  11. def alert: Alerter

    Permalink
    Attributes
    protected
    Definition Classes
    AnyFunSuiteLike → Alerting
  12. def appID: String

    Permalink
    Definition Classes
    SparkContextProvider
  13. def approxEquals(r1: Row, r2: Row, tol: Double): Boolean

    Permalink
    Definition Classes
    DataFrameSuiteBaseLike
  14. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any
  15. def assert[U](message: String, expected: U, actual: U)(implicit CT: ClassTag[U]): Unit

    Permalink
    Definition Classes
    TestSuiteTestSuiteLike
  16. def assert[U](expected: U, actual: U)(implicit CT: ClassTag[U]): Unit

    Permalink
    Definition Classes
    TestSuiteTestSuiteLike
  17. macro def assert(condition: Boolean, clue: Any)(implicit prettifier: Prettifier, pos: Position): Assertion

    Permalink
    Definition Classes
    Assertions
  18. macro def assert(condition: Boolean)(implicit prettifier: Prettifier, pos: Position): Assertion

    Permalink
    Definition Classes
    Assertions
  19. macro def assertCompiles(code: String)(implicit pos: Position): Assertion

    Permalink
    Definition Classes
    Assertions
  20. def assertDataFrameApproximateEquals(expected: DataFrame, result: DataFrame, tol: Double): Unit

    Permalink

    Compares if two DataFrames are equal, checks that the schemas are the same.

    Compares if two DataFrames are equal, checks that the schemas are the same. When comparing inexact fields uses tol.

    tol

    max acceptable tolerance, should be less than 1.

    Definition Classes
    DataFrameSuiteBaseLike
  21. def assertDataFrameDataEquals(expected: DataFrame, result: DataFrame): Unit

    Permalink

    Compares if two DataFrames are equal without caring about order of rows, by finding elements in one DataFrame that is not in the other.

    Compares if two DataFrames are equal without caring about order of rows, by finding elements in one DataFrame that is not in the other. The resulting DataFrame should be empty inferring the two DataFrames have the same elements. Does not compare the schema.

    Definition Classes
    DataFrameSuiteBaseLike
  22. def assertDataFrameEquals(expected: DataFrame, result: DataFrame): Unit

    Permalink

    Compares if two DataFrames are equal, checks the schema and then if that matches checks if the rows are equal.

    Compares if two DataFrames are equal, checks the schema and then if that matches checks if the rows are equal.

    Definition Classes
    DataFrameSuiteBaseLike
  23. def assertDataFrameNoOrderEquals(expected: DataFrame, result: DataFrame): Unit

    Permalink

    Compares if two DataFrames are equal without caring about order of rows, by finding elements in one DataFrame that is not in the other.

    Compares if two DataFrames are equal without caring about order of rows, by finding elements in one DataFrame that is not in the other. The resulting DataFrame should be empty inferring the two DataFrames have the same elements. Also verifies that the schema is identical.

    Definition Classes
    DataFrameSuiteBaseLike
  24. macro def assertDoesNotCompile(code: String)(implicit pos: Position): Assertion

    Permalink
    Definition Classes
    Assertions
  25. def assertEmpty[U](arr: Array[U])(implicit CT: ClassTag[U]): Unit

    Permalink
    Definition Classes
    TestSuiteTestSuiteLike
  26. def assertResult(expected: Any)(actual: Any)(implicit prettifier: Prettifier, pos: Position): Assertion

    Permalink
    Definition Classes
    Assertions
  27. def assertResult(expected: Any, clue: Any)(actual: Any)(implicit prettifier: Prettifier, pos: Position): Assertion

    Permalink
    Definition Classes
    Assertions
  28. def assertSmallDataFrameDataEquals(expected: DataFrame, result: DataFrame): Unit

    Permalink

    Compares if two DataFrames are equal without caring about order of rows, by finding elements in one DataFrame that is not in the other.

    Compares if two DataFrames are equal without caring about order of rows, by finding elements in one DataFrame that is not in the other. Similar to the function assertDataFrameDataEquals but for small DataFrame that can be collected in memory for the comparison.

    Definition Classes
    DataFrameSuiteBaseLike
  29. def assertThrows[T <: AnyRef](f: ⇒ Any)(implicit classTag: ClassTag[T], pos: Position): Assertion

    Permalink
    Definition Classes
    Assertions
  30. def assertTrue(expected: Boolean): Unit

    Permalink
    Definition Classes
    TestSuiteTestSuiteLike
  31. macro def assertTypeError(code: String)(implicit pos: Position): Assertion

    Permalink
    Definition Classes
    Assertions
  32. macro def assume(condition: Boolean, clue: Any)(implicit prettifier: Prettifier, pos: Position): Assertion

    Permalink
    Definition Classes
    Assertions
  33. macro def assume(condition: Boolean)(implicit prettifier: Prettifier, pos: Position): Assertion

    Permalink
    Definition Classes
    Assertions
  34. def beforeAll(): Unit

    Permalink
    Definition Classes
    DataFrameSuiteBaseSharedSparkContext → BeforeAndAfterAll
  35. def builder(): Builder

    Permalink

    Constructs a configuration for hive or iceberg, where the metastore is located in a temp directory.

    Constructs a configuration for hive or iceberg, where the metastore is located in a temp directory.

    Definition Classes
    DataFrameSuiteBaseLike
  36. def cancel(cause: Throwable)(implicit pos: Position): Nothing

    Permalink
    Definition Classes
    Assertions
  37. def cancel(message: String, cause: Throwable)(implicit pos: Position): Nothing

    Permalink
    Definition Classes
    Assertions
  38. def cancel(message: String)(implicit pos: Position): Nothing

    Permalink
    Definition Classes
    Assertions
  39. def cancel()(implicit pos: Position): Nothing

    Permalink
    Definition Classes
    Assertions
  40. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @HotSpotIntrinsicCandidate() @throws( ... )
  41. def conf: SparkConf

    Permalink
    Definition Classes
    SparkContextProvider
  42. def convertEquivalenceToAToBConstraint[A, B](equivalenceOfB: Equivalence[B])(implicit ev: <:<[A, B]): CanEqual[A, B]

    Permalink
    Definition Classes
    TripleEquals → TripleEqualsSupport
  43. def convertEquivalenceToBToAConstraint[A, B](equivalenceOfA: Equivalence[A])(implicit ev: <:<[B, A]): CanEqual[A, B]

    Permalink
    Definition Classes
    TripleEquals → TripleEqualsSupport
  44. def convertToCheckingEqualizer[T](left: T): CheckingEqualizer[T]

    Permalink
    Definition Classes
    TripleEquals → TripleEqualsSupport
  45. implicit def convertToEqualizer[T](left: T): Equalizer[T]

    Permalink
    Definition Classes
    TripleEquals → TripleEqualsSupport
  46. def defaultEquality[A]: Equality[A]

    Permalink
    Definition Classes
    TripleEqualsSupport
  47. def enableHiveSupport: Boolean

    Permalink
    Attributes
    protected
    Definition Classes
    DataFrameSuiteBaseLike
  48. def enableIcebergSupport: Boolean

    Permalink
    Attributes
    protected
    Definition Classes
    DataFrameSuiteBaseLike
  49. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  50. def equals(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  51. final def execute(testName: String, configMap: ConfigMap, color: Boolean, durations: Boolean, shortstacks: Boolean, fullstacks: Boolean, stats: Boolean): Unit

    Permalink
    Definition Classes
    Suite
  52. def expectedTestCount(filter: Filter): Int

    Permalink
    Definition Classes
    Suite
  53. def fail(message: String): Unit

    Permalink
    Definition Classes
    TestSuiteTestSuiteLike
  54. def fail(cause: Throwable)(implicit pos: Position): Nothing

    Permalink
    Definition Classes
    Assertions
  55. def fail(message: String, cause: Throwable)(implicit pos: Position): Nothing

    Permalink
    Definition Classes
    Assertions
  56. def fail(message: String)(implicit pos: Position): Nothing

    Permalink
    Definition Classes
    Assertions
  57. def fail()(implicit pos: Position): Nothing

    Permalink
    Definition Classes
    Assertions
  58. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any
    Annotations
    @HotSpotIntrinsicCandidate()
  59. def hashCode(): Int

    Permalink
    Definition Classes
    AnyRef → Any
    Annotations
    @HotSpotIntrinsicCandidate()
  60. val icebergWarehouse: String

    Permalink
    Definition Classes
    DataFrameSuiteBaseLike
  61. def ignore(testName: String, testTags: Tag*)(testFun: ⇒ Any)(implicit pos: Position): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    AnyFunSuiteLike
  62. implicit def impSqlContext: SQLContext

    Permalink
    Attributes
    protected
    Definition Classes
    DataFrameSuiteBaseLike
  63. def info: Informer

    Permalink
    Attributes
    protected
    Definition Classes
    AnyFunSuiteLike → Informing
  64. def intercept[T <: AnyRef](f: ⇒ Any)(implicit classTag: ClassTag[T], pos: Position): T

    Permalink
    Definition Classes
    Assertions
  65. val invokeBeforeAllAndAfterAllEvenIfNoTestsAreExpected: Boolean

    Permalink
    Definition Classes
    BeforeAndAfterAll
  66. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any
  67. lazy val localMetastorePath: String

    Permalink
    Definition Classes
    DataFrameSuiteBaseLike
  68. lazy val localWarehousePath: String

    Permalink
    Definition Classes
    DataFrameSuiteBaseLike
  69. def lowPriorityTypeCheckedConstraint[A, B](implicit equivalenceOfB: Equivalence[B], ev: <:<[A, B]): CanEqual[A, B]

    Permalink
    Definition Classes
    TripleEquals → TripleEqualsSupport
  70. def markup: Documenter

    Permalink
    Attributes
    protected
    Definition Classes
    AnyFunSuiteLike → Documenting
  71. val maxUnequalRowsToShow: Int

    Permalink
    Definition Classes
    DataFrameSuiteBaseLike
  72. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  73. def nestedSuites: IndexedSeq[Suite]

    Permalink
    Definition Classes
    Suite
  74. def note: Notifier

    Permalink
    Attributes
    protected
    Definition Classes
    AnyFunSuiteLike → Notifying
  75. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @HotSpotIntrinsicCandidate()
  76. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @HotSpotIntrinsicCandidate()
  77. def pending: Assertion with PendingStatement

    Permalink
    Definition Classes
    Assertions
  78. def pendingUntilFixed(f: ⇒ Unit)(implicit pos: Position): Assertion with PendingStatement

    Permalink
    Definition Classes
    Assertions
  79. final def registerIgnoredTest(testText: String, testTags: Tag*)(testFun: ⇒ Any)(implicit pos: Position): Unit

    Permalink
    Definition Classes
    AnyFunSuiteLike → TestRegistration
  80. final def registerTest(testText: String, testTags: Tag*)(testFun: ⇒ Any)(implicit pos: Position): Unit

    Permalink
    Definition Classes
    AnyFunSuiteLike → TestRegistration
  81. def rerunner: Option[String]

    Permalink
    Definition Classes
    Suite
  82. implicit def reuseContextIfPossible: Boolean

    Permalink
    Attributes
    protected
    Definition Classes
    SharedSparkContext
  83. def run(testName: Option[String], args: Args): Status

    Permalink
    Definition Classes
    BeforeAndAfterAll → SuiteMixin
  84. def runNestedSuites(args: Args): Status

    Permalink
    Attributes
    protected
    Definition Classes
    Suite
  85. def runTest(testName: String, args: Args): Status

    Permalink
    Attributes
    protected
    Definition Classes
    AnyFunSuiteLike → TestSuite → Suite
  86. def runTests(testName: Option[String], args: Args): Status

    Permalink
    Attributes
    protected
    Definition Classes
    AnyFunSuiteLike → Suite
  87. def sc: SparkContext

    Permalink
  88. def setup(sc: SparkContext): Unit

    Permalink

    Setup work to be called when creating a new SparkContext.

    Setup work to be called when creating a new SparkContext. Default implementation currently sets a checkpoint directory.

    This _should_ be called by the context provider automatically.

    Definition Classes
    SparkContextProvider
  89. lazy val spark: SparkSession

    Permalink
    Definition Classes
    DataFrameSuiteBaseLike
  90. def sqlBeforeAllTestCases(): Unit

    Permalink
    Definition Classes
    DataFrameSuiteBaseLike
  91. lazy val sqlContext: SQLContext

    Permalink
    Definition Classes
    DataFrameSuiteBaseLike
  92. final val succeed: Assertion

    Permalink
    Definition Classes
    Assertions
  93. def suiteId: String

    Permalink
    Definition Classes
    Suite
  94. def suiteName: String

    Permalink
    Definition Classes
    Suite
  95. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef
  96. def tags: Map[String, Set[String]]

    Permalink
    Definition Classes
    AnyFunSuiteLike → Suite
  97. lazy val tempDir: File

    Permalink
    Definition Classes
    DataFrameSuiteBaseLike
  98. def test(testName: String, testTags: Tag*)(testFun: ⇒ Any)(implicit pos: Position): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    AnyFunSuiteLike
  99. def testCodegenOnly(testName: String, testTags: Tag*)(testFun: ⇒ Any)(implicit pos: Position): Unit

    Permalink
  100. def testCombined(testName: String, testTags: Tag*)(testFun: ⇒ Any)(implicit pos: Position): Unit

    Permalink
  101. def testDataFor(testName: String, theConfigMap: ConfigMap): TestData

    Permalink
    Definition Classes
    AnyFunSuiteLike → Suite
  102. def testNames: Set[String]

    Permalink
    Definition Classes
    AnyFunSuiteLike → Suite
  103. def testNonCodegen(testName: String, testTags: Tag*)(testFun: ⇒ Any)(implicit pos: Position): Unit

    Permalink
  104. def testsFor(unit: Unit): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    AnyFunSuiteLike
  105. def toString(): String

    Permalink
    Definition Classes
    AnyFunSuite → AnyFunSuiteLike → AnyRef → Any
  106. def typeCheckedConstraint[A, B](implicit equivalenceOfA: Equivalence[A], ev: <:<[B, A]): CanEqual[A, B]

    Permalink
    Definition Classes
    TripleEquals → TripleEqualsSupport
  107. implicit def unconstrainedEquality[A, B](implicit equalityOfA: Equality[A]): CanEqual[A, B]

    Permalink
    Definition Classes
    TripleEquals → TripleEqualsSupport
  108. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  109. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  110. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  111. def withClue[T](clue: Any)(fun: ⇒ T): T

    Permalink
    Definition Classes
    Assertions
  112. def withFixture(test: NoArgTest): Outcome

    Permalink
    Attributes
    protected
    Definition Classes
    TestSuite
  113. def withSQLConf(pairs: (String, String)*)(f: ⇒ Unit): Unit

    Permalink

    Sets all SQL configurations specified in pairs, calls f, and then restores all SQL configurations.

    Sets all SQL configurations specified in pairs, calls f, and then restores all SQL configurations. Taken from Spark SQLHelper.

    Attributes
    protected
    Definition Classes
    DataFrameSuiteBase

Deprecated Value Members

  1. def conversionCheckedConstraint[A, B](implicit equivalenceOfA: Equivalence[A], cnv: (B) ⇒ A): CanEqual[A, B]

    Permalink
    Definition Classes
    TripleEquals → TripleEqualsSupport
    Annotations
    @deprecated
    Deprecated

    (Since version 3.1.0) The conversionCheckedConstraint method has been deprecated and will be removed in a future version of ScalaTest. It is no longer needed now that the deprecation period of ConversionCheckedTripleEquals has expired. It will not be replaced.

  2. def convertEquivalenceToAToBConversionConstraint[A, B](equivalenceOfB: Equivalence[B])(implicit ev: (A) ⇒ B): CanEqual[A, B]

    Permalink
    Definition Classes
    TripleEquals → TripleEqualsSupport
    Annotations
    @deprecated
    Deprecated

    (Since version 3.1.0) The convertEquivalenceToAToBConversionConstraint method has been deprecated and will be removed in a future version of ScalaTest. It is no longer needed now that the deprecation period of ConversionCheckedTripleEquals has expired. It will not be replaced.

  3. def convertEquivalenceToBToAConversionConstraint[A, B](equivalenceOfA: Equivalence[A])(implicit ev: (B) ⇒ A): CanEqual[A, B]

    Permalink
    Definition Classes
    TripleEquals → TripleEqualsSupport
    Annotations
    @deprecated
    Deprecated

    (Since version 3.1.0) The convertEquivalenceToBToAConversionConstraint method has been deprecated and will be removed in a future version of ScalaTest. It is no longer needed now that the deprecation period of ConversionCheckedTripleEquals has expired. It will not be replaced.

  4. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @Deprecated @deprecated @throws( classOf[java.lang.Throwable] )
    Deprecated

    (Since version ) see corresponding Javadoc for more information.

  5. def lowPriorityConversionCheckedConstraint[A, B](implicit equivalenceOfB: Equivalence[B], cnv: (A) ⇒ B): CanEqual[A, B]

    Permalink
    Definition Classes
    TripleEquals → TripleEqualsSupport
    Annotations
    @deprecated
    Deprecated

    (Since version 3.1.0) The lowPriorityConversionCheckedConstraint method has been deprecated and will be removed in a future version of ScalaTest. It is no longer needed now that the deprecation period of ConversionCheckedTripleEquals has expired. It will not be replaced.

  6. final val styleName: String

    Permalink
    Definition Classes
    AnyFunSuiteLike → Suite
    Annotations
    @deprecated
    Deprecated

    (Since version 3.1.0) The styleName lifecycle method has been deprecated and will be removed in a future version of ScalaTest with no replacement.

Inherited from DataFrameSuiteBase

Inherited from DataFrameSuiteBaseLike

Inherited from SharedSparkContext

Inherited from SparkContextProvider

Inherited from BeforeAndAfterAll

Inherited from SuiteMixin

Inherited from TestSuite

Inherited from TestSuiteLike

Inherited from AnyFunSuite

Inherited from AnyFunSuiteLike

Inherited from Documenting

Inherited from Alerting

Inherited from Notifying

Inherited from Informing

Inherited from TestRegistration

Inherited from org.scalatest.TestSuite

Inherited from Suite

Inherited from Serializable

Inherited from Serializable

Inherited from Assertions

Inherited from TripleEquals

Inherited from TripleEqualsSupport

Inherited from AnyRef

Inherited from Any

Ungrouped