Package

org.apache.spark

memory

Permalink

package memory

This package implements Spark's memory management system. This system consists of two main components, a JVM-wide memory manager and a per-task manager:

Internally, each of these components have additional abstractions for memory bookkeeping:

Diagrammatically:

+-------------+
| MemConsumer |----+                                   +------------------------+
+-------------+    |    +-------------------+          |     MemoryManager      |
                   +--->| TaskMemoryManager |----+     |                        |
+-------------+    |    +-------------------+    |     |  +------------------+  |
| MemConsumer |----+                             |     |  |  StorageMemPool  |  |
+-------------+         +-------------------+    |     |  +------------------+  |
                        | TaskMemoryManager |----+     |                        |
                        +-------------------+    |     |  +------------------+  |
                                                 +---->|  |OnHeapExecMemPool |  |
                                 *               |     |  +------------------+  |
                                 *               |     |                        |
+-------------+                  *               |     |  +------------------+  |
| MemConsumer |----+                             |     |  |OffHeapExecMemPool|  |
+-------------+    |    +-------------------+    |     |  +------------------+  |
                   +--->| TaskMemoryManager |----+     |                        |
                        +-------------------+          +------------------------+

There are two implementations of org.apache.spark.memory.MemoryManager which vary in how they handle the sizing of their memory pools:

Linear Supertypes
AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. memory
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Type Members

  1. abstract class MemoryConsumer extends AnyRef

    Permalink
  2. final class MemoryMode extends Enum[MemoryMode]

    Permalink
  3. class TaskMemoryManager extends AnyRef

    Permalink

Value Members

  1. object UnifiedMemoryManager

    Permalink

Inherited from AnyRef

Inherited from Any

Ungrouped