KafkaConsume
Value members
Methods
- Note
- you have to first use
subscribe
to subscribe the consumer
before using thisStream
. If you forgot to subscribe, there
will be a NotSubscribedException raised in theStream
.
Stream
where the elements themselves are Stream
s which continuallyrequest records for a single partition. These
Stream
s will have to beprocessed in parallel, using
parJoin
or parJoinUnbounded
. Note thatwhen using
parJoin(n)
and n
is smaller than the number of currentlyassigned partitions, then there will be assigned partitions which won't
be processed. For that reason, prefer
parJoinUnbounded
and the actuallimit will be the number of assigned partitions.
If you do not want to process all partitions in parallel, then you
can use stream instead, where records for all partitions are in
a single
Stream
.- Note
- you have to first use
subscribe
to subscribe the consumer
before using thisStream
. If you forgot to subscribe, there
will be a NotSubscribedException raised in theStream
.
def partitionsMapStream: Stream[F, Map[TopicPartition, Stream[F, CommittableConsumerRecord[F, K, V]]]]
Stream
where each element contains a current assignment. The currentassignment is the
Map
, where keys is a TopicPartition
, and values arestreams with records for a particular
TopicPartition
.New assignments will be received on each rebalance. On rebalance,
Kafka revoke all previously assigned partitions, and after that assigned
new partitions all at once.
partitionsMapStream
reflects this processin a streaming manner.
Note, that partition streams for revoked partitions will
be closed after the new assignment comes.
This is the most generic
Stream
method. If you don't need such control,consider using
partitionedStream
or stream
methods.They are both based on a
partitionsMapStream
.- See also
- Note
- you have to first use
subscribe
to subscribe the consumer
before using thisStream
. If you forgot to subscribe, there
will be a NotSubscribedException raised in theStream
.
Stops consuming new messages from Kafka.
This method could be used to implement a graceful shutdown.
This method has a few effects:
1. After this call no more data will be fetched from Kafka through the
2. All currently running streams will continue to run until all in-flight messages will be processed.
It means that streams will be completed when all fetched messages will be processed.
If some of the stream methods will be called after stopConsuming call,
these methods will return empty streams.
More than one call of stopConsuming will have no effect.
This method could be used to implement a graceful shutdown.
This method has a few effects:
1. After this call no more data will be fetched from Kafka through the
poll
method.2. All currently running streams will continue to run until all in-flight messages will be processed.
It means that streams will be completed when all fetched messages will be processed.
If some of the stream methods will be called after stopConsuming call,
these methods will return empty streams.
More than one call of stopConsuming will have no effect.