Package org.apache.parquet.hadoop

Class MemoryManager

  • public class MemoryManager
extends Object
    Implements a memory manager that keeps a global context of how many Parquet writers there are and manages the memory between them. For use cases with dynamic partitions, it is easy to end up with many writers in the same task. By managing the size of each allocation, we try to cut down the size of each allocation and keep the task from running out of memory. This class balances the allocation size of each writer by resize them averagely. When the sum of each writer's allocation size is less than total memory pool, keep their original value. When the sum exceeds, decrease each writer's allocation size by a ratio.

    • Constructor Detail

      • MemoryManager

        public MemoryManager​(float ratio,
                     long minAllocation)

    • Method Detail

      • registerScaleCallBack

        public void registerScaleCallBack​(String callBackName,
                                  Runnable callBack)
        Register callback and deduplicate it if any.
        Parameters:
        callBackName - the name of callback. It should be identical.
        callBack - the callback passed in from upper layer, such as Hive.