Package org.apache.flink.runtime.operators.hash

Class HashPartition<BT,​PT>

  • Type Parameters:
    BT - The type of the build side records.
    PT - The type of the probe side records.
    All Implemented Interfaces:
    DataInput, org.apache.flink.core.memory.DataInputView, org.apache.flink.core.memory.SeekableDataInputView
    Direct Known Subclasses:
    ReOpenableHashPartition
    public class HashPartition<BT,​PT>
extends AbstractPagedInputView
implements org.apache.flink.core.memory.SeekableDataInputView

    • Field Detail

      • overflowSegments

        protected org.apache.flink.core.memory.MemorySegment[] overflowSegments

      • numOverflowSegments

        protected int numOverflowSegments

      • nextOverflowBucket

        protected int nextOverflowBucket

      • partitionBuffers

        protected org.apache.flink.core.memory.MemorySegment[] partitionBuffers

      • probeSideRecordCounter

        protected long probeSideRecordCounter

      • recursionLevel

        protected int recursionLevel

      • probeSideChannel

        protected BlockChannelWriter<org.apache.flink.core.memory.MemorySegment> probeSideChannel

      • furtherPartitioning

        protected boolean furtherPartitioning

    • Method Detail

      • setFurtherPatitioning

        protected void setFurtherPatitioning​(boolean v)

      • getPartitionNumber

        public int getPartitionNumber()
        Gets the partition number of this partition.
        Returns:
        This partition's number.

      • getRecursionLevel

        public int getRecursionLevel()
        Gets this partition's recursion level.
        Returns:
        The partition's recursion level.

      • isInMemory

        public final boolean isInMemory()
        Checks whether this partition is in memory or spilled.
        Returns:
        True, if the partition is in memory, false if it is spilled.

      • getNumOccupiedMemorySegments

        public int getNumOccupiedMemorySegments()
        Gets the number of memory segments used by this partition, which includes build side memory buffers and overflow memory segments.
        Returns:
        The number of occupied memory segments.

      • getBuildSideBlockCount

        public int getBuildSideBlockCount()

      • getProbeSideBlockCount

        public int getProbeSideBlockCount()

      • getBuildSideRecordCount

        public long getBuildSideRecordCount()

      • getProbeSideRecordCount

        public long getProbeSideRecordCount()

      • getBuildSideChannel

        public BlockChannelWriter<org.apache.flink.core.memory.MemorySegment> getBuildSideChannel()

      • getProbeSideChannel

        public BlockChannelWriter<org.apache.flink.core.memory.MemorySegment> getProbeSideChannel()

      • insertIntoBuildBuffer

        public final long insertIntoBuildBuffer​(BT record)
                                 throws IOException
        Inserts the given object into the current buffer. This method returns a pointer that can be used to address the written record in this partition, if it is in-memory. The returned pointers have no expressiveness in the case where the partition is spilled.
        Parameters:
        record - The object to be written to the partition.
        Returns:
        A pointer to the object in the partition, or -1, if the partition is spilled.
        Throws:
        IOException - Thrown, when this is a spilled partition and the write failed.

      • insertIntoProbeBuffer

        public final void insertIntoProbeBuffer​(PT record)
                                 throws IOException
        Inserts the given record into the probe side buffers. This method is only applicable when the partition was spilled while processing the build side.

        If this method is invoked when the partition is still being built, it has undefined behavior.

        Parameters:
        record - The record to be inserted into the probe side buffers.
        Throws:
        IOException - Thrown, if the buffer is full, needs to be spilled, and spilling causes an error.

      • spillPartition

        public int spillPartition​(List<org.apache.flink.core.memory.MemorySegment> target,
                          IOManager ioAccess,
                          FileIOChannel.ID targetChannel,
                          LinkedBlockingQueue<org.apache.flink.core.memory.MemorySegment> bufferReturnQueue)
                   throws IOException
        Spills this partition to disk and sets it up such that it continues spilling records that are added to it. The spilling process must free at least one buffer, either in the partition's record buffers, or in the memory segments for overflow buckets. The partition immediately takes back one buffer to use it for further spilling.
        Parameters:
        target - The list to which memory segments from overflow buckets are added.
        ioAccess - The I/O manager to be used to create a writer to disk.
        targetChannel - The id of the target channel for this partition.
        Returns:
        The number of buffers that were freed by spilling this partition.
        Throws:
        IOException - Thrown, if the writing failed.

      • finalizeProbePhase

        public int finalizeProbePhase​(List<org.apache.flink.core.memory.MemorySegment> freeMemory,
                              List<HashPartition<BT,​PT>> spilledPartitions,
                              boolean keepUnprobedSpilledPartitions)
                       throws IOException
        Parameters:
        keepUnprobedSpilledPartitions - If true then partitions that were spilled but received no further probe requests will be retained; used for build-side outer joins.
        Returns:
        The number of write-behind buffers reclaimable after this method call.
        Throws:
        IOException

      • clearAllMemory

        public void clearAllMemory​(List<org.apache.flink.core.memory.MemorySegment> target)

      • setReadPosition

        public void setReadPosition​(long pointer)
        Specified by:
        setReadPosition in interface org.apache.flink.core.memory.SeekableDataInputView

      • nextSegment

        protected org.apache.flink.core.memory.MemorySegment nextSegment​(org.apache.flink.core.memory.MemorySegment current)
                                                          throws IOException
        Description copied from class: AbstractPagedInputView
        The method by which concrete subclasses realize page crossing. This method is invoked when the current page is exhausted and a new page is required to continue the reading. If no further page is available, this method must throw an EOFException.
        Specified by:
        nextSegment in class AbstractPagedInputView
        Parameters:
        current - The current page that was read to its limit. May be null, if this method is invoked for the first time.
        Returns:
        The next page from which the reading should continue. May not be null. If the input is exhausted, an EOFException must be thrown instead.
        Throws:
        EOFException - Thrown, if no further segment is available.
        IOException - Thrown, if the method cannot provide the next page due to an I/O related problem.

      • getLimitForSegment

        protected int getLimitForSegment​(org.apache.flink.core.memory.MemorySegment segment)
        Description copied from class: AbstractPagedInputView
        Gets the limit for reading bytes from the given memory segment. This method must return the position of the byte after the last valid byte in the given memory segment. When the position returned by this method is reached, the view will attempt to switch to the next memory segment.
        Specified by:
        getLimitForSegment in class AbstractPagedInputView
        Parameters:
        segment - The segment to determine the limit for.
        Returns:
        The limit for the given memory segment.