Mailing-List: contact user-help@storm.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@storm.apache.org
MIME-Version: 1.0
In-Reply-To: <CAMdZAAKo9_4x8zPi8PF_=ojkufoVrnuWSrgxNe6V38tsVtrgZw@mail.gmail.com>
References: <CAMdZAAKo9_4x8zPi8PF_=ojkufoVrnuWSrgxNe6V38tsVtrgZw@mail.gmail.com>
From: =?UTF-8?Q?Stig_Rohde_D=C3=B8ssing?= <srdo@apache.org>
Date: Tue, 7 Nov 2017 07:52:42 +0100
Message-ID: <CAG09ER24RXRMj1W7Ef4ic7e=Virj6jAdpdUMd1-ozV3jVV8vpg@mail.gmail.com>
Subject: Re: KafkaSpout not committing to some kafka partitions and stop consuming
To: user@storm.apache.org
Content-Type: multipart/alternative; boundary="94eb2c1305908ae7a1055d5f048a"
archived-at: Tue, 07 Nov 2017 06:52:46 -0000

--94eb2c1305908ae7a1055d5f048a
Content-Type: text/plain; charset="UTF-8"

Hi,

There are a few bugs related to this in 1.1.0. One cause could be that the
message is right, and you have some failed tuples that haven't been
committed which are preventing the spout from progressing. You could also
be hitting either of the issues fixed in
https://github.com/apache/storm/pull/2156 and
https://github.com/apache/storm/pull/2307/files. The second fix is only
relevant if you're using topic compaction in Kafka.

You might want to try checking out and building storm-kafka-client from one
of these branches and see if that fixes your issue. You should be able to
use the upgraded storm-kafka-client with your current Storm version.

2017-11-07 4:44 GMT+01:00 Zhechao Ma <mazhechaomaillist@gmail.com>:

> Hi
>
> We are using KafkaSpout to read messages from kafka servers. We've
> frequently come across the situation that KafkaSpout stops consuming from
> some kafka partitions. We check logs and find the following log.
>
> 2017-11-07 11:35:03.170 o.a.s.k.s.KafkaSpout Thread-11-spout-executor[121
> 121] [DEBUG] Not polling. [10000083] uncommitted offsets across all topic
> partitions has reached the threshold of [10000000]
>
> When it reaches to the maxUncommittedOffsets (10000000), KafkaSpout stops
> committing to
> those partitions, and never consumes again. We also find that this issue
> usually happens after partition reassignment.
>
> Storm 1.0.2
> storm-kafka-clients 1.1.0
> Kafka 0.10.1.1
>
> Could anyone give me help?
>
> --
> Thanks
> Zhechao Ma
>

--94eb2c1305908ae7a1055d5f048a
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div><div>Hi,<br><br></div>There are a few bugs related to=
 this in 1.1.0. One cause could be that the message is right, and you have =
some failed tuples that haven&#39;t been committed which are preventing the=
 spout from progressing. You could also be hitting either of the issues fix=
ed in <a href=3D"https://github.com/apache/storm/pull/2156">https://github.=
com/apache/storm/pull/2156</a> and <a href=3D"https://github.com/apache/sto=
rm/pull/2307/files">https://github.com/apache/storm/pull/2307/files</a>. Th=
e second fix is only relevant if you&#39;re using topic compaction in Kafka=
.<br><br></div>You might want to try checking out and building storm-kafka-=
client from one of these branches and see if that fixes your issue. You sho=
uld be able to use the upgraded storm-kafka-client with your current Storm =
version.<br></div><div class=3D"gmail_extra"><br><div class=3D"gmail_quote"=
>2017-11-07 4:44 GMT+01:00 Zhechao Ma <span dir=3D"ltr">&lt;<a href=3D"mail=
to:mazhechaomaillist@gmail.com" target=3D"_blank">mazhechaomaillist@gmail.c=
om</a>&gt;</span>:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0=
 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr"><div>=
<div>Hi<br><br></div>We are using KafkaSpout to read messages from kafka se=
rvers. We&#39;ve frequently come across the situation that KafkaSpout stops=
 consuming from some kafka partitions. We check logs and find the following=
 log.<br></div><div><br></div><div>2017-11-07 11:35:03.170 o.a.s.k.s.KafkaS=
pout Thread-11-spout-executor[121 121] [DEBUG] Not polling. [10000083] unco=
mmitted offsets across all topic partitions has reached the threshold of [1=
0000000]<br></div><div><br></div>When it reaches to the maxUncommittedOffse=
ts (10000000), KafkaSpout stops committing to <br clear=3D"all"><div><div>t=
hose partitions, and never consumes again. We also find that this issue usu=
ally happens after partition reassignment.<br></div><div><br></div><div>Sto=
rm 1.0.2</div><div>storm-kafka-clients 1.1.0</div><div>Kafka 0.10.1.1</div>=
<div><br></div><div>Could anyone give me help?<span class=3D"HOEnZb"><font =
color=3D"#888888"><br></font></span></div><span class=3D"HOEnZb"><font colo=
r=3D"#888888"><div><br></div><div>-- <br><div class=3D"m_288527995836493009=
2gmail_signature"><div dir=3D"ltr">Thanks<br>Zhechao Ma<br></div></div>
</div></font></span></div></div>
</blockquote></div><br></div>

--94eb2c1305908ae7a1055d5f048a--