incubator-kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tim Lossen <...@lossen.de>
Subject Re: corrupted topic
Date Wed, 26 Oct 2011 07:28:45 GMT
neha,

>>> it looked as if some of them use the same logic as the ruby client, and might
also be affected.
> 
> Could you please list the clients that you think might have the same bug ?


well, the c#, go and php client seem all to be doing more or
less the same thing as the ruby client, as far as i can tell ;)

when consuming a batch of messages, the last thing in the 
response buffer will often be an incomplete message. the logic 
to detect and skip this was broken, which results in a subsequent
parsing error.

cheers
tim


On 2011-10-25, at 19:17 , Neha Narkhede wrote:

> Tim,
> 
> Thanks for looking into this !
> 
>>> it looked as if some of them use the same logic as the ruby client, and might
also be affected.
> 
> Could you please list the clients that you think might have the same bug ?
> 
>> are you actually using any client besides the java / scala one in production at linkedin?
> 
> No.
> 
> Thanks,
> Neha
> 
> On Tue, Oct 25, 2011 at 9:16 AM, Tim Lossen <tim@lossen.de> wrote:
>> jun,
>> 
>> the ruby client (maintained by alejandro crosa) is here:
>> 
>> https://github.com/acrosa/kafka-rb
>> 
>> i noticed just now that he also seems to work at linkedin?
>> 
>> last commit on github is a bugfix, on october 14. but there
>> also seem to be some changes on the apache side which are not
>> in the github version ... maybe alejandro can best sort this
>> out himself.
>> 
>> while looking over the other clients when we where hunting for
>> this bug, it looked as if some of them use the same logic as
>> the ruby client, and might also be affected.
>> 
>> are you actually using any client besides the java / scala one
>> in production at linkedin?
>> 
>> cheers
>> tim
>> 
>> 
>> On 2011-10-25, at 17:58 , Jun Rao wrote:
>> 
>>> Tim,
>>> 
>>> Thanks for the update. What's the github url for the ruby client? Has it
>>> diverged from what's in Apache?
>>> 
>>> I agree with you that we should consider excluding those clients not well
>>> maintained from our distribution.
>>> 
>>> Jun
>>> 
>>> On Tue, Oct 25, 2011 at 7:46 AM, Tim Lossen <tim@lossen.de> wrote:
>>> 
>>>> ok, we finally traced this issue to a bug in the ruby kafka client,
>>>> which we were able to fix -- the topic was never corrupted.
>>>> 
>>>> (we sent a pull request to the maintainer of the client on github.)
>>>> 
>>>> BTW, i do not think that it is a good idea to include an (outdated)
>>>> copy of the ruby client (and other clients) in the kafka distribution.
>>>> maybe better *link* to the actual client projects?
>>>> 
>>>> cheers
>>>> tim
>>>> 
>>>> On 2011-10-24, at 21:30 , Tim Lossen wrote:
>>>> 
>>>>> ok, thanks -- tomorrow we'll try investigate further ...
>>>>> 
>>>>> 
>>>>> On 2011-10-24, at 9:12 PM, Neha Narkhede wrote:
>>>>> 
>>>>>> Tim,
>>>>>> 
>>>>>>> what if the CRC32 checksum is correct, but the internal binary
message
>>>> structure is not?
>>>>>> 
>>>>>> The CRC check involves recomputing the CRC and then checking against
>>>>>> the stored CRC in the header. The probability of that matching is
>>>>>> extremely low.
>>>>>> 
>>>>>> Corruption is also possible if the broker crashes in the middle of
a
>>>>>> flush. In that case, when the broker restarts, it detects an unclean
>>>>>> shutdown, runs recovery on the logs and truncates the log if the
CRC
>>>>>> check fails at some message.
>>>>>> 
>>>>>> Also, we compute the CRC only on the payload of the message. So
>>>>>> technically, some bits could get flipped in the header of the message.
>>>>>> 
>>>>>> Thanks,
>>>>>> Neha
>>>>>> 
>>>>>> On Mon, Oct 24, 2011 at 12:07 PM, Tim Lossen <tim@lossen.de>
wrote:
>>>>>>> what if the CRC32 checksum is correct, but the internal binary
message
>>>>>>> structure is not?
>>>>>>> 
>>>>>>> 
>>>>>>> On 2011-10-24, at 8:56 PM, Jay Kreps wrote:
>>>>>>> 
>>>>>>>> It is not supposed to be possible. We include a CRC32 with
each
>>>> message,
>>>>>>>> so
>>>>>>>> invalid requests should be detected and rejected. But that
does not
>>>>>>>> preclude
>>>>>>>> the possibility that we missed a case.
>>>>>>>> 
>>>>>>>> -Jay
>>>>>>>> 
>>>>>>>> On Mon, Oct 24, 2011 at 11:41 AM, Tim Lossen <tim@lossen.de>
wrote:
>>>>>>>> 
>>>>>>>>> hi,
>>>>>>>>> 
>>>>>>>>> is it possible for a faulty client to "corrupt" a topic
on the
>>>> broker,
>>>>>>>>> so that consumers cannot consume messages any more? or
does
>>>>>>>>> the broker protect itself against this?
>>>>>>>>> 
>>>>>>>>> i am asking because we seem to have run into such a situation.
>>>>>>>>> we are using a perl producer and a ruby consumer. the
per lib might
>>>>>>>>> be a bit outdated.
>>>>>>>>> 
>>>>>>>>> cheers
>>>>>>>>> tim
>>>>>>>>> 
>>>>>>>>> --
>>>>>>>>> http://tim.lossen.de
>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>>> --
>>>>>>> http://tim.lossen.de
>>>>>>> 
>>>>>>> 
>>>>> 
>>>>> --
>>>>> http://tim.lossen.de
>>>>> 
>>>> 
>>>> --
>>>> http://tim.lossen.de
>>>> 
>>>> 
>>>> 
>>>> 
>> 
>> --
>> http://tim.lossen.de
>> 
>> 
>> 
>> 

--
http://tim.lossen.de




Mime
View raw message