incubator-kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tim Lossen <...@lossen.de>
Subject Re: corrupted topic
Date Mon, 24 Oct 2011 19:30:30 GMT
ok, thanks -- tomorrow we'll try investigate further ...


On 2011-10-24, at 9:12 PM, Neha Narkhede wrote:

> Tim,
>
>> what if the CRC32 checksum is correct, but the internal binary  
>> message structure is not?
>
> The CRC check involves recomputing the CRC and then checking against
> the stored CRC in the header. The probability of that matching is
> extremely low.
>
> Corruption is also possible if the broker crashes in the middle of a
> flush. In that case, when the broker restarts, it detects an unclean
> shutdown, runs recovery on the logs and truncates the log if the CRC
> check fails at some message.
>
> Also, we compute the CRC only on the payload of the message. So
> technically, some bits could get flipped in the header of the message.
>
> Thanks,
> Neha
>
> On Mon, Oct 24, 2011 at 12:07 PM, Tim Lossen <tim@lossen.de> wrote:
>> what if the CRC32 checksum is correct, but the internal binary  
>> message
>> structure is not?
>>
>>
>> On 2011-10-24, at 8:56 PM, Jay Kreps wrote:
>>
>>> It is not supposed to be possible. We include a CRC32 with each  
>>> message,
>>> so
>>> invalid requests should be detected and rejected. But that does not
>>> preclude
>>> the possibility that we missed a case.
>>>
>>> -Jay
>>>
>>> On Mon, Oct 24, 2011 at 11:41 AM, Tim Lossen <tim@lossen.de> wrote:
>>>
>>>> hi,
>>>>
>>>> is it possible for a faulty client to "corrupt" a topic on the  
>>>> broker,
>>>> so that consumers cannot consume messages any more? or does
>>>> the broker protect itself against this?
>>>>
>>>> i am asking because we seem to have run into such a situation.
>>>> we are using a perl producer and a ruby consumer. the per lib might
>>>> be a bit outdated.
>>>>
>>>> cheers
>>>> tim
>>>>
>>>> --
>>>> http://tim.lossen.de
>>>>
>>>>
>>
>> --
>> http://tim.lossen.de
>>
>>

--
http://tim.lossen.de


Mime
View raw message