org.apache.spark.sql.execution.aggregate

TungstenAggregationIterator

class TungstenAggregationIterator extends AggregationIterator with Logging

An iterator used to evaluate aggregate functions. It operates on UnsafeRows.

This iterator first uses hash-based aggregation to process input rows. It uses a hash map to store groups and their corresponding aggregation buffers. If this map cannot allocate memory from memory manager, it spills the map into disk and creates a new one. After processed all the input, then merge all the spills together using external sorter, and do sort-based aggregation.

The process has the following step:

The code of this class is organized as follows:

Linear Supertypes
AggregationIterator, Logging, Iterator[UnsafeRow], TraversableOnce[UnsafeRow], GenTraversableOnce[UnsafeRow], AnyRef, Any
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. TungstenAggregationIterator
  2. AggregationIterator
  3. Logging
  4. Iterator
  5. TraversableOnce
  6. GenTraversableOnce
  7. AnyRef
  8. Any
  1. Hide All
  2. Show all
Learn more about member selection
Visibility
  1. Public
  2. All

Instance Constructors

  1. new TungstenAggregationIterator(groupingExpressions: Seq[NamedExpression], aggregateExpressions: Seq[AggregateExpression], aggregateAttributes: Seq[Attribute], initialInputBufferOffset: Int, resultExpressions: Seq[NamedExpression], newMutableProjection: (Seq[Expression], Seq[Attribute]) ⇒ MutableProjection, originalInputAttributes: Seq[Attribute], inputIter: Iterator[InternalRow], testFallbackStartsAt: Option[(Int, Int)], numOutputRows: SQLMetric, peakMemory: SQLMetric, spillSize: SQLMetric)

    groupingExpressions

    expressions for grouping keys

    aggregateExpressions

    AggregateExpression containing AggregateFunctions with mode Partial, PartialMerge, or Final.

    aggregateAttributes

    the attributes of the aggregateExpressions' outputs when they are stored in the final aggregation buffer.

    resultExpressions

    expressions for generating output rows.

    newMutableProjection

    the function used to create mutable projections.

    originalInputAttributes

    attributes of representing input rows from inputIter.

    inputIter

    the iterator containing input UnsafeRows.

Type Members

  1. class GroupedIterator[B >: A] extends AbstractIterator[Seq[B]] with Iterator[Seq[B]]

    Definition Classes
    Iterator

Value Members

  1. final def !=(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  2. final def !=(arg0: Any): Boolean

    Definition Classes
    Any

  3. final def ##(): Int

    Definition Classes
    AnyRef → Any

  4. def ++[B >: UnsafeRow](that: ⇒ GenTraversableOnce[B]): Iterator[B]

    Definition Classes
    Iterator

  5. def /:[B](z: B)(op: (B, UnsafeRow) ⇒ B): B

    Definition Classes
    TraversableOnce → GenTraversableOnce

  6. def :\[B](z: B)(op: (UnsafeRow, B) ⇒ B): B

    Definition Classes
    TraversableOnce → GenTraversableOnce

  7. final def ==(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  8. final def ==(arg0: Any): Boolean

    Definition Classes
    Any

  9. def addString(b: StringBuilder): StringBuilder

    Definition Classes
    TraversableOnce

  10. def addString(b: StringBuilder, sep: String): StringBuilder

    Definition Classes
    TraversableOnce

  11. def addString(b: StringBuilder, start: String, sep: String, end: String): StringBuilder

    Definition Classes
    TraversableOnce

  12. def aggregate[B](z: B)(seqop: (B, UnsafeRow) ⇒ B, combop: (B, B) ⇒ B): B

    Definition Classes
    TraversableOnce → GenTraversableOnce

  13. val aggregateFunctions: Array[AggregateFunction]

    Attributes
    protected
    Definition Classes
    AggregationIterator

  14. val allImperativeAggregateFunctionPositions: Array[Int]

    Attributes
    protected[this]
    Definition Classes
    AggregationIterator

  15. val allImperativeAggregateFunctions: Array[ImperativeAggregate]

    Attributes
    protected[this]
    Definition Classes
    AggregationIterator

  16. final def asInstanceOf[T0]: T0

    Definition Classes
    Any

  17. def buffered: BufferedIterator[UnsafeRow]

    Definition Classes
    Iterator

  18. def clone(): AnyRef

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  19. def collect[B](pf: PartialFunction[UnsafeRow, B]): Iterator[B]

    Definition Classes
    Iterator
    Annotations
    @migration
    Migration

    (Changed in version 2.8.0) collect has changed. The previous behavior can be reproduced with toSeq.

  20. def collectFirst[B](pf: PartialFunction[UnsafeRow, B]): Option[B]

    Definition Classes
    TraversableOnce

  21. def contains(elem: Any): Boolean

    Definition Classes
    Iterator

  22. def copyToArray[B >: UnsafeRow](xs: Array[B], start: Int, len: Int): Unit

    Definition Classes
    Iterator → TraversableOnce → GenTraversableOnce

  23. def copyToArray[B >: UnsafeRow](xs: Array[B]): Unit

    Definition Classes
    TraversableOnce → GenTraversableOnce

  24. def copyToArray[B >: UnsafeRow](xs: Array[B], start: Int): Unit

    Definition Classes
    TraversableOnce → GenTraversableOnce

  25. def copyToBuffer[B >: UnsafeRow](dest: Buffer[B]): Unit

    Definition Classes
    TraversableOnce

  26. def corresponds[B](that: GenTraversableOnce[B])(p: (UnsafeRow, B) ⇒ Boolean): Boolean

    Definition Classes
    Iterator

  27. def count(p: (UnsafeRow) ⇒ Boolean): Int

    Definition Classes
    TraversableOnce → GenTraversableOnce

  28. def drop(n: Int): Iterator[UnsafeRow]

    Definition Classes
    Iterator

  29. def dropWhile(p: (UnsafeRow) ⇒ Boolean): Iterator[UnsafeRow]

    Definition Classes
    Iterator

  30. def duplicate: (Iterator[UnsafeRow], Iterator[UnsafeRow])

    Definition Classes
    Iterator

  31. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  32. def equals(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any

  33. def exists(p: (UnsafeRow) ⇒ Boolean): Boolean

    Definition Classes
    Iterator → TraversableOnce → GenTraversableOnce

  34. val expressionAggInitialProjection: MutableProjection

    Attributes
    protected[this]
    Definition Classes
    AggregationIterator

  35. def filter(p: (UnsafeRow) ⇒ Boolean): Iterator[UnsafeRow]

    Definition Classes
    Iterator

  36. def filterNot(p: (UnsafeRow) ⇒ Boolean): Iterator[UnsafeRow]

    Definition Classes
    Iterator

  37. def finalize(): Unit

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )

  38. def find(p: (UnsafeRow) ⇒ Boolean): Option[UnsafeRow]

    Definition Classes
    Iterator → TraversableOnce → GenTraversableOnce

  39. def flatMap[B](f: (UnsafeRow) ⇒ GenTraversableOnce[B]): Iterator[B]

    Definition Classes
    Iterator

  40. def fold[A1 >: UnsafeRow](z: A1)(op: (A1, A1) ⇒ A1): A1

    Definition Classes
    TraversableOnce → GenTraversableOnce

  41. def foldLeft[B](z: B)(op: (B, UnsafeRow) ⇒ B): B

    Definition Classes
    TraversableOnce → GenTraversableOnce

  42. def foldRight[B](z: B)(op: (UnsafeRow, B) ⇒ B): B

    Definition Classes
    TraversableOnce → GenTraversableOnce

  43. def forall(p: (UnsafeRow) ⇒ Boolean): Boolean

    Definition Classes
    Iterator → TraversableOnce → GenTraversableOnce

  44. def foreach[U](f: (UnsafeRow) ⇒ U): Unit

    Definition Classes
    Iterator → TraversableOnce → GenTraversableOnce

  45. val generateOutput: (UnsafeRow, InternalRow) ⇒ UnsafeRow

    Attributes
    protected
    Definition Classes
    AggregationIterator

  46. def generateProcessRow(expressions: Seq[AggregateExpression], functions: Seq[AggregateFunction], inputAttributes: Seq[Attribute]): (InternalRow, InternalRow) ⇒ Unit

    Attributes
    protected
    Definition Classes
    AggregationIterator

  47. def generateResultProjection(): (UnsafeRow, InternalRow) ⇒ UnsafeRow

    Attributes
    protected
    Definition Classes
    TungstenAggregationIterator → AggregationIterator

  48. final def getClass(): Class[_]

    Definition Classes
    AnyRef → Any

  49. def grouped[B >: UnsafeRow](size: Int): GroupedIterator[B]

    Definition Classes
    Iterator

  50. val groupingAttributes: Seq[Attribute]

    Attributes
    protected
    Definition Classes
    AggregationIterator

  51. val groupingProjection: UnsafeProjection

    Attributes
    protected
    Definition Classes
    AggregationIterator

  52. def hasDefiniteSize: Boolean

    Definition Classes
    Iterator → TraversableOnce → GenTraversableOnce

  53. final def hasNext: Boolean

    Definition Classes
    TungstenAggregationIterator → Iterator

  54. def hashCode(): Int

    Definition Classes
    AnyRef → Any

  55. def indexOf[B >: UnsafeRow](elem: B): Int

    Definition Classes
    Iterator

  56. def indexWhere(p: (UnsafeRow) ⇒ Boolean): Int

    Definition Classes
    Iterator

  57. def initializeAggregateFunctions(expressions: Seq[AggregateExpression], startingInputBufferOffset: Int): Array[AggregateFunction]

    Attributes
    protected
    Definition Classes
    AggregationIterator

  58. def initializeBuffer(buffer: InternalRow): Unit

    Initializes buffer values for all aggregate functions.

    Initializes buffer values for all aggregate functions.

    Attributes
    protected
    Definition Classes
    AggregationIterator

  59. def initializeLogIfNecessary(isInterpreter: Boolean): Unit

    Attributes
    protected
    Definition Classes
    Logging

  60. def isEmpty: Boolean

    Definition Classes
    Iterator → TraversableOnce → GenTraversableOnce

  61. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any

  62. def isTraceEnabled(): Boolean

    Attributes
    protected
    Definition Classes
    Logging

  63. def isTraversableAgain: Boolean

    Definition Classes
    Iterator → GenTraversableOnce

  64. def length: Int

    Definition Classes
    Iterator

  65. def log: Logger

    Attributes
    protected
    Definition Classes
    Logging

  66. def logDebug(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging

  67. def logDebug(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging

  68. def logError(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging

  69. def logError(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging

  70. def logInfo(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging

  71. def logInfo(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging

  72. def logName: String

    Attributes
    protected
    Definition Classes
    Logging

  73. def logTrace(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging

  74. def logTrace(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging

  75. def logWarning(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging

  76. def logWarning(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging

  77. def map[B](f: (UnsafeRow) ⇒ B): Iterator[B]

    Definition Classes
    Iterator

  78. def max[B >: UnsafeRow](implicit cmp: Ordering[B]): UnsafeRow

    Definition Classes
    TraversableOnce → GenTraversableOnce

  79. def maxBy[B](f: (UnsafeRow) ⇒ B)(implicit cmp: Ordering[B]): UnsafeRow

    Definition Classes
    TraversableOnce → GenTraversableOnce

  80. def min[B >: UnsafeRow](implicit cmp: Ordering[B]): UnsafeRow

    Definition Classes
    TraversableOnce → GenTraversableOnce

  81. def minBy[B](f: (UnsafeRow) ⇒ B)(implicit cmp: Ordering[B]): UnsafeRow

    Definition Classes
    TraversableOnce → GenTraversableOnce

  82. def mkString: String

    Definition Classes
    TraversableOnce → GenTraversableOnce

  83. def mkString(sep: String): String

    Definition Classes
    TraversableOnce → GenTraversableOnce

  84. def mkString(start: String, sep: String, end: String): String

    Definition Classes
    TraversableOnce → GenTraversableOnce

  85. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  86. final def next(): UnsafeRow

    Definition Classes
    TungstenAggregationIterator → Iterator

  87. def nonEmpty: Boolean

    Definition Classes
    TraversableOnce → GenTraversableOnce

  88. final def notify(): Unit

    Definition Classes
    AnyRef

  89. final def notifyAll(): Unit

    Definition Classes
    AnyRef

  90. def outputForEmptyGroupingKeyWithoutInput(): UnsafeRow

    Generate an output row when there is no input and there is no grouping expression.

  91. def padTo[A1 >: UnsafeRow](len: Int, elem: A1): Iterator[A1]

    Definition Classes
    Iterator

  92. def partition(p: (UnsafeRow) ⇒ Boolean): (Iterator[UnsafeRow], Iterator[UnsafeRow])

    Definition Classes
    Iterator

  93. def patch[B >: UnsafeRow](from: Int, patchElems: Iterator[B], replaced: Int): Iterator[B]

    Definition Classes
    Iterator

  94. val processRow: (InternalRow, InternalRow) ⇒ Unit

    Attributes
    protected
    Definition Classes
    AggregationIterator

  95. def product[B >: UnsafeRow](implicit num: Numeric[B]): B

    Definition Classes
    TraversableOnce → GenTraversableOnce

  96. def reduce[A1 >: UnsafeRow](op: (A1, A1) ⇒ A1): A1

    Definition Classes
    TraversableOnce → GenTraversableOnce

  97. def reduceLeft[B >: UnsafeRow](op: (B, UnsafeRow) ⇒ B): B

    Definition Classes
    TraversableOnce

  98. def reduceLeftOption[B >: UnsafeRow](op: (B, UnsafeRow) ⇒ B): Option[B]

    Definition Classes
    TraversableOnce → GenTraversableOnce

  99. def reduceOption[A1 >: UnsafeRow](op: (A1, A1) ⇒ A1): Option[A1]

    Definition Classes
    TraversableOnce → GenTraversableOnce

  100. def reduceRight[B >: UnsafeRow](op: (UnsafeRow, B) ⇒ B): B

    Definition Classes
    TraversableOnce → GenTraversableOnce

  101. def reduceRightOption[B >: UnsafeRow](op: (UnsafeRow, B) ⇒ B): Option[B]

    Definition Classes
    TraversableOnce → GenTraversableOnce

  102. def reversed: List[UnsafeRow]

    Attributes
    protected[this]
    Definition Classes
    TraversableOnce

  103. def sameElements(that: Iterator[_]): Boolean

    Definition Classes
    Iterator

  104. def scanLeft[B](z: B)(op: (B, UnsafeRow) ⇒ B): Iterator[B]

    Definition Classes
    Iterator

  105. def scanRight[B](z: B)(op: (UnsafeRow, B) ⇒ B): Iterator[B]

    Definition Classes
    Iterator

  106. def seq: Iterator[UnsafeRow]

    Definition Classes
    Iterator → TraversableOnce → GenTraversableOnce

  107. def size: Int

    Definition Classes
    TraversableOnce → GenTraversableOnce

  108. def slice(from: Int, until: Int): Iterator[UnsafeRow]

    Definition Classes
    Iterator

  109. def sliding[B >: UnsafeRow](size: Int, step: Int): GroupedIterator[B]

    Definition Classes
    Iterator

  110. def span(p: (UnsafeRow) ⇒ Boolean): (Iterator[UnsafeRow], Iterator[UnsafeRow])

    Definition Classes
    Iterator

  111. def sum[B >: UnsafeRow](implicit num: Numeric[B]): B

    Definition Classes
    TraversableOnce → GenTraversableOnce

  112. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef

  113. def take(n: Int): Iterator[UnsafeRow]

    Definition Classes
    Iterator

  114. def takeWhile(p: (UnsafeRow) ⇒ Boolean): Iterator[UnsafeRow]

    Definition Classes
    Iterator

  115. def to[Col[_]](implicit cbf: CanBuildFrom[Nothing, UnsafeRow, Col[UnsafeRow]]): Col[UnsafeRow]

    Definition Classes
    TraversableOnce → GenTraversableOnce

  116. def toArray[B >: UnsafeRow](implicit arg0: ClassTag[B]): Array[B]

    Definition Classes
    TraversableOnce → GenTraversableOnce

  117. def toBuffer[B >: UnsafeRow]: Buffer[B]

    Definition Classes
    TraversableOnce → GenTraversableOnce

  118. def toIndexedSeq: IndexedSeq[UnsafeRow]

    Definition Classes
    TraversableOnce → GenTraversableOnce

  119. def toIterable: Iterable[UnsafeRow]

    Definition Classes
    TraversableOnce → GenTraversableOnce

  120. def toIterator: Iterator[UnsafeRow]

    Definition Classes
    Iterator → GenTraversableOnce

  121. def toList: List[UnsafeRow]

    Definition Classes
    TraversableOnce → GenTraversableOnce

  122. def toMap[T, U](implicit ev: <:<[UnsafeRow, (T, U)]): Map[T, U]

    Definition Classes
    TraversableOnce → GenTraversableOnce

  123. def toSeq: Seq[UnsafeRow]

    Definition Classes
    TraversableOnce → GenTraversableOnce

  124. def toSet[B >: UnsafeRow]: Set[B]

    Definition Classes
    TraversableOnce → GenTraversableOnce

  125. def toStream: Stream[UnsafeRow]

    Definition Classes
    Iterator → GenTraversableOnce

  126. def toString(): String

    Definition Classes
    Iterator → AnyRef → Any

  127. def toTraversable: Traversable[UnsafeRow]

    Definition Classes
    Iterator → TraversableOnce → GenTraversableOnce

  128. def toVector: Vector[UnsafeRow]

    Definition Classes
    TraversableOnce → GenTraversableOnce

  129. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  130. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  131. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  132. def withFilter(p: (UnsafeRow) ⇒ Boolean): Iterator[UnsafeRow]

    Definition Classes
    Iterator

  133. def zip[B](that: Iterator[B]): Iterator[(UnsafeRow, B)]

    Definition Classes
    Iterator

  134. def zipAll[B, A1 >: UnsafeRow, B1 >: B](that: Iterator[B], thisElem: A1, thatElem: B1): Iterator[(A1, B1)]

    Definition Classes
    Iterator

  135. def zipWithIndex: Iterator[(UnsafeRow, Int)]

    Definition Classes
    Iterator

Deprecated Value Members

  1. def /:\[A1 >: UnsafeRow](z: A1)(op: (A1, A1) ⇒ A1): A1

    Definition Classes
    GenTraversableOnce
    Annotations
    @deprecated
    Deprecated

    (Since version 2.10.0) use fold instead

Inherited from AggregationIterator

Inherited from Logging

Inherited from Iterator[UnsafeRow]

Inherited from TraversableOnce[UnsafeRow]

Inherited from GenTraversableOnce[UnsafeRow]

Inherited from AnyRef

Inherited from Any

Ungrouped