storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stig Rohde Døssing <s...@apache.org>
Subject Re: KafkaSpout not committing to some kafka partitions and stop consuming
Date Tue, 07 Nov 2017 06:52:42 GMT
Hi,

There are a few bugs related to this in 1.1.0. One cause could be that the
message is right, and you have some failed tuples that haven't been
committed which are preventing the spout from progressing. You could also
be hitting either of the issues fixed in
https://github.com/apache/storm/pull/2156 and
https://github.com/apache/storm/pull/2307/files. The second fix is only
relevant if you're using topic compaction in Kafka.

You might want to try checking out and building storm-kafka-client from one
of these branches and see if that fixes your issue. You should be able to
use the upgraded storm-kafka-client with your current Storm version.

2017-11-07 4:44 GMT+01:00 Zhechao Ma <mazhechaomaillist@gmail.com>:

> Hi
>
> We are using KafkaSpout to read messages from kafka servers. We've
> frequently come across the situation that KafkaSpout stops consuming from
> some kafka partitions. We check logs and find the following log.
>
> 2017-11-07 11:35:03.170 o.a.s.k.s.KafkaSpout Thread-11-spout-executor[121
> 121] [DEBUG] Not polling. [10000083] uncommitted offsets across all topic
> partitions has reached the threshold of [10000000]
>
> When it reaches to the maxUncommittedOffsets (10000000), KafkaSpout stops
> committing to
> those partitions, and never consumes again. We also find that this issue
> usually happens after partition reassignment.
>
> Storm 1.0.2
> storm-kafka-clients 1.1.0
> Kafka 0.10.1.1
>
> Could anyone give me help?
>
> --
> Thanks
> Zhechao Ma
>

Mime
View raw message