KafkaConsume
Attributes
- Source
- KafkaConsume.scala
- Graph
-
- Supertypes
- Known subtypes
-
Members list
Value members
Abstract methods
Stream
where the elements themselves are Stream
s which continually request records for a single partition.
Stream
where the elements themselves are Stream
s which continually request records for a single partition. These Stream
s will have to be processed in parallel, using parJoin
or parJoinUnbounded
. Note that when using parJoin(n)
and n
is smaller than the number of currently assigned partitions, then there will be assigned partitions which won't be processed. For that reason, prefer parJoinUnbounded
and the actual limit will be the number of assigned partitions.
If you do not want to process all partitions in parallel, then you can use records instead, where records for all partitions are in a single Stream
.
Attributes
- Note
-
you have to first use
subscribe
orassign
the consumer before using thisStream
. If you forgot to subscribe, there will be a NotSubscribedException raised in theStream
. - Source
- KafkaConsume.scala
Stream
where each element contains a Map
with all newly assigned partitions.
Stream
where each element contains a Map
with all newly assigned partitions. Keys of this Map
are TopicPartition
s, and values are record streams for the particular TopicPartition
. These streams will be closed only when a partition is revoked.
With the default assignor, all previous partitions are revoked at once, and a new set of partitions is assigned to a consumer on each rebalance. In this case, each returned Map
contains the full partition assignment for the consumer. And all streams from the previous assignment are closed. It means, that partitionsMapStream
reflects the default assignment process in a streaming manner.
This may not be the case when a custom assignor is configured in the consumer. When using the CooperativeStickyAssignor
, for instance, partitions may be revoked individually. In this case, each element in the stream (eachMap
) will contain only streams for newly assigned partitions. Previously returned streams for partitions that are retained will remain active. Only streams for revoked partitions will be closed.
This is the most generic Stream
method. If you don't need such control, consider using partitionedStream
or stream
methods. They are both based on a partitionsMapStream
.
Attributes
- See also
- Note
-
you have to first use
subscribe
orassign
to subscribe the consumer before using thisStream
. If you forgot to subscribe, there will be a NotSubscribedException raised in theStream
. - Source
- KafkaConsume.scala
Stops consuming new messages from Kafka.
Stops consuming new messages from Kafka. This method could be used to implement a graceful shutdown.
This method has a few effects: 1. After this call no more data will be fetched from Kafka through the poll
method. 2. All currently running streams will continue to run until all in-flight messages will be processed. It means that streams will be completed when all fetched messages will be processed.
If some of the records methods will be called after stopConsuming call, these methods will return empty streams.
More than one call of stopConsuming will have no effect.
Attributes
- Source
- KafkaConsume.scala
Alias for partitionedStream.parJoinUnbounded
.
Alias for partitionedStream.parJoinUnbounded
. See partitionedRecords for more information.
Attributes
- Note
-
you have to first use
subscribe
orassign
the consumer before using thisStream
. If you forgot to subscribe, there will be a NotSubscribedException raised in theStream
. - Source
- KafkaConsume.scala
Concrete methods
Alias for partitionedStream
Consume from all assigned partitions, producing a stream of CommittableConsumerRecords.
Consume from all assigned partitions, producing a stream of CommittableConsumerRecords. Alias for stream.
Attributes
- Source
- KafkaConsume.scala