Mailing-List: contact kafka-users-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: kafka-users@incubator.apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1084)
Subject: Re: corrupted topic
From: Tim Lossen <tim@lossen.de>
In-Reply-To: 
 <CAOG_4QZ1OCYZiQxW8PxjXub2Q61Styrw+V2HbryL7HET5VMFkA@mail.gmail.com>
Date: Wed, 26 Oct 2011 09:28:45 +0200
Content-Transfer-Encoding: quoted-printable
Message-Id: <61F214B3-6A0D-48BE-AFFC-367FD6E9BC25@lossen.de>
References: <EA3037B3-9B3D-4790-A191-CA3D1BBA4E7E@lossen.de>
 <CAOeJiJjx5uwWec31JJ+nMOv13uMg+tDmY_+VseF=2eFycAATPg@mail.gmail.com>
 <B25B8324-E73D-4EE2-BC77-9A3A14B90D5E@lossen.de>
 <CAOG_4QYU2rOyKxuJDU-4g3VCr=MutuAsTfko03R5LdHxgmEEuQ@mail.gmail.com>
 <C3782997-BEAB-4910-8C3B-8A973A989278@lossen.de>
 <0D7DCA44-2FDB-467B-821B-466F28CE7218@lossen.de>
 <CAFbh0Q3Lw7F5t7D16F9nhoMGngTMH1RH64SjHJ-qkDbKzY=oAg@mail.gmail.com>
 <8E42288F-D38D-42DF-8395-DAB540E5F133@lossen.de>
 <CAOG_4QZ1OCYZiQxW8PxjXub2Q61Styrw+V2HbryL7HET5VMFkA@mail.gmail.com>
To: kafka-users@incubator.apache.org

neha,

>>> it looked as if some of them use the same logic as the ruby client, =
and might also be affected.
>=20
> Could you please list the clients that you think might have the same =
bug ?


well, the c#, go and php client seem all to be doing more or
less the same thing as the ruby client, as far as i can tell ;)

when consuming a batch of messages, the last thing in the=20
response buffer will often be an incomplete message. the logic=20
to detect and skip this was broken, which results in a subsequent
parsing error.

cheers
tim


On 2011-10-25, at 19:17 , Neha Narkhede wrote:

> Tim,
>=20
> Thanks for looking into this !
>=20
>>> it looked as if some of them use the same logic as the ruby client, =
and might also be affected.
>=20
> Could you please list the clients that you think might have the same =
bug ?
>=20
>> are you actually using any client besides the java / scala one in =
production at linkedin?
>=20
> No.
>=20
> Thanks,
> Neha
>=20
> On Tue, Oct 25, 2011 at 9:16 AM, Tim Lossen <tim@lossen.de> wrote:
>> jun,
>>=20
>> the ruby client (maintained by alejandro crosa) is here:
>>=20
>> https://github.com/acrosa/kafka-rb
>>=20
>> i noticed just now that he also seems to work at linkedin?
>>=20
>> last commit on github is a bugfix, on october 14. but there
>> also seem to be some changes on the apache side which are not
>> in the github version ... maybe alejandro can best sort this
>> out himself.
>>=20
>> while looking over the other clients when we where hunting for
>> this bug, it looked as if some of them use the same logic as
>> the ruby client, and might also be affected.
>>=20
>> are you actually using any client besides the java / scala one
>> in production at linkedin?
>>=20
>> cheers
>> tim
>>=20
>>=20
>> On 2011-10-25, at 17:58 , Jun Rao wrote:
>>=20
>>> Tim,
>>>=20
>>> Thanks for the update. What's the github url for the ruby client? =
Has it
>>> diverged from what's in Apache?
>>>=20
>>> I agree with you that we should consider excluding those clients not =
well
>>> maintained from our distribution.
>>>=20
>>> Jun
>>>=20
>>> On Tue, Oct 25, 2011 at 7:46 AM, Tim Lossen <tim@lossen.de> wrote:
>>>=20
>>>> ok, we finally traced this issue to a bug in the ruby kafka client,
>>>> which we were able to fix -- the topic was never corrupted.
>>>>=20
>>>> (we sent a pull request to the maintainer of the client on github.)
>>>>=20
>>>> BTW, i do not think that it is a good idea to include an (outdated)
>>>> copy of the ruby client (and other clients) in the kafka =
distribution.
>>>> maybe better *link* to the actual client projects?
>>>>=20
>>>> cheers
>>>> tim
>>>>=20
>>>> On 2011-10-24, at 21:30 , Tim Lossen wrote:
>>>>=20
>>>>> ok, thanks -- tomorrow we'll try investigate further ...
>>>>>=20
>>>>>=20
>>>>> On 2011-10-24, at 9:12 PM, Neha Narkhede wrote:
>>>>>=20
>>>>>> Tim,
>>>>>>=20
>>>>>>> what if the CRC32 checksum is correct, but the internal binary =
message
>>>> structure is not?
>>>>>>=20
>>>>>> The CRC check involves recomputing the CRC and then checking =
against
>>>>>> the stored CRC in the header. The probability of that matching is
>>>>>> extremely low.
>>>>>>=20
>>>>>> Corruption is also possible if the broker crashes in the middle =
of a
>>>>>> flush. In that case, when the broker restarts, it detects an =
unclean
>>>>>> shutdown, runs recovery on the logs and truncates the log if the =
CRC
>>>>>> check fails at some message.
>>>>>>=20
>>>>>> Also, we compute the CRC only on the payload of the message. So
>>>>>> technically, some bits could get flipped in the header of the =
message.
>>>>>>=20
>>>>>> Thanks,
>>>>>> Neha
>>>>>>=20
>>>>>> On Mon, Oct 24, 2011 at 12:07 PM, Tim Lossen <tim@lossen.de> =
wrote:
>>>>>>> what if the CRC32 checksum is correct, but the internal binary =
message
>>>>>>> structure is not?
>>>>>>>=20
>>>>>>>=20
>>>>>>> On 2011-10-24, at 8:56 PM, Jay Kreps wrote:
>>>>>>>=20
>>>>>>>> It is not supposed to be possible. We include a CRC32 with each
>>>> message,
>>>>>>>> so
>>>>>>>> invalid requests should be detected and rejected. But that does =
not
>>>>>>>> preclude
>>>>>>>> the possibility that we missed a case.
>>>>>>>>=20
>>>>>>>> -Jay
>>>>>>>>=20
>>>>>>>> On Mon, Oct 24, 2011 at 11:41 AM, Tim Lossen <tim@lossen.de> =
wrote:
>>>>>>>>=20
>>>>>>>>> hi,
>>>>>>>>>=20
>>>>>>>>> is it possible for a faulty client to "corrupt" a topic on the
>>>> broker,
>>>>>>>>> so that consumers cannot consume messages any more? or does
>>>>>>>>> the broker protect itself against this?
>>>>>>>>>=20
>>>>>>>>> i am asking because we seem to have run into such a situation.
>>>>>>>>> we are using a perl producer and a ruby consumer. the per lib =
might
>>>>>>>>> be a bit outdated.
>>>>>>>>>=20
>>>>>>>>> cheers
>>>>>>>>> tim
>>>>>>>>>=20
>>>>>>>>> --
>>>>>>>>> http://tim.lossen.de
>>>>>>>>>=20
>>>>>>>>>=20
>>>>>>>=20
>>>>>>> --
>>>>>>> http://tim.lossen.de
>>>>>>>=20
>>>>>>>=20
>>>>>=20
>>>>> --
>>>>> http://tim.lossen.de
>>>>>=20
>>>>=20
>>>> --
>>>> http://tim.lossen.de
>>>>=20
>>>>=20
>>>>=20
>>>>=20
>>=20
>> --
>> http://tim.lossen.de
>>=20
>>=20
>>=20
>>=20

--
http://tim.lossen.de