Package io.confluent.parallelconsumer.internal

Class AbstractParallelEoSStreamProcessor<K,V>

java.lang.Object
io.confluent.parallelconsumer.internal.AbstractParallelEoSStreamProcessor<K,V>
All Implemented Interfaces:
DrainingCloseable, ParallelConsumer<K,V>, Closeable, AutoCloseable, org.apache.kafka.clients.consumer.ConsumerRebalanceListener
Direct Known Subclasses:
ExternalEngine, ParallelEoSStreamProcessor
public abstract class AbstractParallelEoSStreamProcessor<K,V> extends Object implements ParallelConsumer<K,V>, org.apache.kafka.clients.consumer.ConsumerRebalanceListener, Closeable
See Also:

  • Field Details

  • Constructor Details

    • AbstractParallelEoSStreamProcessor

      protected AbstractParallelEoSStreamProcessor(ParallelConsumerOptions<K,V> newOptions)

    • AbstractParallelEoSStreamProcessor

      protected AbstractParallelEoSStreamProcessor(ParallelConsumerOptions<K,V> newOptions, PCModule<K,V> module)
      Construct the AsyncConsumer by wrapping this passed in conusmer and producer, which can be configured any which way as per normal.
      See Also:

  • Method Details

    • setTimeBetweenCommits

      @Deprecated public void setTimeBetweenCommits(Duration timeBetweenCommits)
      Deprecated.
      use ParallelConsumerOptions.ParallelConsumerOptionsBuilder.commitInterval(java.time.Duration)} instead. This will be deleted in the next major version.
      Sets the time between commits. Using a higher frequency will put more load on the brokers.

    • getTimeBetweenCommits

      @Deprecated public Duration getTimeBetweenCommits()
      Deprecated.
      use ParallelConsumerOptions.setCommitInterval(java.time.Duration) instead. This will be deleted in the next major version.
      Gets the time between commits.

    • isClosedOrFailed

      public boolean isClosedOrFailed()

    • getFailureCause

      public Exception getFailureCause()
      Returns:
      if the system failed, returns the recorded reason.

    • setupWorkerPool

      protected ThreadPoolExecutor setupWorkerPool(int poolSize)

    • subscribe

      public void subscribe(Collection<String> topics)
      Specified by:
      subscribe in interface ParallelConsumer<K,V>
      See Also:
      • KafkaConsumer.subscribe(Collection)

    • subscribe

      public void subscribe(Pattern pattern)
      Specified by:
      subscribe in interface ParallelConsumer<K,V>
      See Also:
      • KafkaConsumer.subscribe(Pattern)

    • subscribe

      public void subscribe(Collection<String> topics, org.apache.kafka.clients.consumer.ConsumerRebalanceListener callback)
      Specified by:
      subscribe in interface ParallelConsumer<K,V>
      See Also:
      • KafkaConsumer.subscribe(Collection, ConsumerRebalanceListener)

    • subscribe

      public void subscribe(Pattern pattern, org.apache.kafka.clients.consumer.ConsumerRebalanceListener callback)
      Specified by:
      subscribe in interface ParallelConsumer<K,V>
      See Also:
      • KafkaConsumer.subscribe(Pattern, ConsumerRebalanceListener)

    • onPartitionsRevoked

      public void onPartitionsRevoked(Collection<org.apache.kafka.common.TopicPartition> partitions)
      Commit our offsets

      Make sure the calling thread is the thread which performs commit - i.e. is the OffsetCommitter.

      Specified by:
      onPartitionsRevoked in interface org.apache.kafka.clients.consumer.ConsumerRebalanceListener

    • onPartitionsAssigned

      public void onPartitionsAssigned(Collection<org.apache.kafka.common.TopicPartition> partitions)
      Delegate to WorkManager
      Specified by:
      onPartitionsAssigned in interface org.apache.kafka.clients.consumer.ConsumerRebalanceListener
      See Also:

    • onPartitionsLost

      public void onPartitionsLost(Collection<org.apache.kafka.common.TopicPartition> partitions)
      Cannot commit any offsets for partitions that have been `lost` (as opposed to revoked). Just delegate to WorkManager for truncation.
      Specified by:
      onPartitionsLost in interface org.apache.kafka.clients.consumer.ConsumerRebalanceListener
      See Also:

    • close

      public void close()
      Close the system, without draining.
      Specified by:
      close in interface AutoCloseable
      Specified by:
      close in interface Closeable
      Specified by:
      close in interface DrainingCloseable
      See Also:

    • close

      public void close(Duration timeout, DrainingCloseable.DrainingMode drainMode)
      Description copied from interface: DrainingCloseable
      Close the consumer.
      Specified by:
      close in interface DrainingCloseable
      Parameters:
      timeout - how long to wait before giving up
      drainMode - wait for messages already consumed from the broker to be processed before closing

    • waitForProcessedNotCommitted

      @Deprecated public void waitForProcessedNotCommitted(Duration timeout)
      Deprecated.
      no longer used, will be removed in next version
      Block the calling thread until no more messages are being processed.

      Used for testing.

    • supervisorLoop

      protected <R> void supervisorLoop(Function<PollContextInternal<K,V>,List<R>> userFunctionWrapped, Consumer<R> callback)
      Kicks off the control loop in the executor, with supervision and returns.
      See Also:

    • controlLoop

      protected <R> void controlLoop(Function<PollContextInternal<K,V>,List<R>> userFunction, Consumer<R> callback) throws TimeoutException, ExecutionException, InterruptedException
      Main control loop
      Throws:
      TimeoutException
      ExecutionException
      InterruptedException

    • submitWorkToPool

      protected <R> void submitWorkToPool(Function<PollContextInternal<K,V>,List<R>> usersFunction, Consumer<R> callback, List<WorkContainer<K,V>> workToProcess)
      Submit a piece of work to the processing pool.
      Parameters:
      workToProcess - the polled records to process

    • calculateQuantityToRequest

      protected int calculateQuantityToRequest()
      Returns:
      number of WorkContainer to try to get

    • getTargetOutForProcessing

      protected int getTargetOutForProcessing()

    • getQueueTargetLoaded

      protected int getQueueTargetLoaded()

    • checkPipelinePressure

      protected void checkPipelinePressure()
      Checks the system has enough pressure in the pipeline of work, if not attempts to step up the load factor.

    • processWorkCompleteMailBox

      protected void processWorkCompleteMailBox(Duration timeToBlockFor)
      Check the work queue for work to be done, potentially blocking.

      Can be interrupted if something else needs doing.

      Visible for testing.

    • isTimeToCommitNow

      protected boolean isTimeToCommitNow()

    • commitOffsetsThatAreReady

      protected void commitOffsetsThatAreReady() throws TimeoutException, InterruptedException
      Visible for testing
      Throws:
      TimeoutException
      InterruptedException

    • runUserFunction

      protected <R> List<ParallelConsumer.Tuple<org.apache.kafka.clients.consumer.ConsumerRecord<K,V>,R>> runUserFunction(Function<PollContextInternal<K,V>,List<R>> usersFunction, Consumer<R> callback, List<WorkContainer<K,V>> workContainerBatch)
      Run the supplied function.

    • addToMailBoxOnUserFunctionSuccess

      protected void addToMailBoxOnUserFunctionSuccess(PollContextInternal<K,V> context, WorkContainer<K,V> wc, List<?> resultsFromUserFunction)

    • onUserFunctionSuccess

      protected void onUserFunctionSuccess(WorkContainer<K,V> wc, List<?> resultsFromUserFunction)

    • addToMailbox

      protected void addToMailbox(PollContextInternal<K,V> pollContext, WorkContainer<K,V> wc)

    • registerWork

      public void registerWork(EpochAndRecordsMap<K,V> polledRecords)

    • notifySomethingToDo

      public void notifySomethingToDo()
      Early notify of work arrived.

      Only wake up the thread if it's sleeping while polling the mail box.

      See Also:

    • workRemaining

      public long workRemaining()
      Description copied from interface: DrainingCloseable
      Of the records consumed from the broker, how many do we have remaining in our local queues
      Specified by:
      workRemaining in interface DrainingCloseable
      Returns:
      the number of consumed but outstanding records to process

    • addLoopEndCallBack

      public void addLoopEndCallBack(Runnable r)
      Plugin a function to run at the end of each main loop.

      Useful for testing and controlling loop progression.

    • setLongPollTimeout

      public void setLongPollTimeout(Duration ofMillis)

    • requestCommitAsap

      public void requestCommitAsap()
      Request a commit as soon as possible (ASAP), overriding other constraints.

    • pauseIfRunning

      public void pauseIfRunning()
      Description copied from interface: ParallelConsumer
      Pause this consumer (i.e. stop processing of messages).

      This operation only has an effect if the consumer is currently running. In all other cases calling this method will be silent a no-op.

      Once the consumer is paused, the system will stop submitting work to the processing pool. Already submitted in flight work however will be finished. This includes work that is currently being processed inside a user function as well as work that has already been submitted to the processing pool but has not been picked up by a free worker yet.

      General remarks:

      • A paused consumer may still keep polling for new work until internal buffers are filled.
      • This operation does not actively pause the subscription on the underlying Kafka Broker (compared to KafkaConsumer#pause).
      • Pending offset commits will still be performed when the consumer is paused.

      Specified by:
      pauseIfRunning in interface ParallelConsumer<K,V>

    • resumeIfPaused

      public void resumeIfPaused()
      Description copied from interface: ParallelConsumer
      Resume this consumer (i.e. continue processing of messages).

      This operation only has an effect if the consumer is currently paused. In all other cases calling this method will be a silent no-op.

      Specified by:
      resumeIfPaused in interface ParallelConsumer<K,V>

    • getOptions

      protected ParallelConsumerOptions getOptions()

    • getProducerManager

      protected Optional<ProducerManager<K,V>> getProducerManager()

    • getWm

      public WorkManager<K,V> getWm()

    • getWorkMailBox

      protected BlockingQueue<io.confluent.parallelconsumer.internal.AbstractParallelEoSStreamProcessor.ControllerEventMessage<K,V>> getWorkMailBox()
      Collection of work waiting to be

    • setState

      public void setState(State state)
      The run state of the controller.
      See Also:

    • getNumberOfAssignedPartitions

      public int getNumberOfAssignedPartitions()

    • setMyId

      public void setMyId(Optional<String> myId)
      Optional ID of this instance. Useful for testing.

    • getMyId

      public Optional<String> getMyId()
      Optional ID of this instance. Useful for testing.