storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Zhechao Ma <mazhechaomaill...@gmail.com>
Subject KafkaSpout not committing to some kafka partitions and stop consuming
Date Tue, 07 Nov 2017 03:44:07 GMT
Hi

We are using KafkaSpout to read messages from kafka servers. We've
frequently come across the situation that KafkaSpout stops consuming from
some kafka partitions. We check logs and find the following log.

2017-11-07 11:35:03.170 o.a.s.k.s.KafkaSpout Thread-11-spout-executor[121
121] [DEBUG] Not polling. [10000083] uncommitted offsets across all topic
partitions has reached the threshold of [10000000]

When it reaches to the maxUncommittedOffsets (10000000), KafkaSpout stops
committing to
those partitions, and never consumes again. We also find that this issue
usually happens after partition reassignment.

Storm 1.0.2
storm-kafka-clients 1.1.0
Kafka 0.10.1.1

Could anyone give me help?

-- 
Thanks
Zhechao Ma

Mime
View raw message