incubator-kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Neha Narkhede <neha.narkh...@gmail.com>
Subject Re: corrupted topic
Date Mon, 24 Oct 2011 19:12:54 GMT
Tim,

> what if the CRC32 checksum is correct, but the internal binary message structure is not?

The CRC check involves recomputing the CRC and then checking against
the stored CRC in the header. The probability of that matching is
extremely low.

Corruption is also possible if the broker crashes in the middle of a
flush. In that case, when the broker restarts, it detects an unclean
shutdown, runs recovery on the logs and truncates the log if the CRC
check fails at some message.

Also, we compute the CRC only on the payload of the message. So
technically, some bits could get flipped in the header of the message.

Thanks,
Neha

On Mon, Oct 24, 2011 at 12:07 PM, Tim Lossen <tim@lossen.de> wrote:
> what if the CRC32 checksum is correct, but the internal binary message
> structure is not?
>
>
> On 2011-10-24, at 8:56 PM, Jay Kreps wrote:
>
>> It is not supposed to be possible. We include a CRC32 with each message,
>> so
>> invalid requests should be detected and rejected. But that does not
>> preclude
>> the possibility that we missed a case.
>>
>> -Jay
>>
>> On Mon, Oct 24, 2011 at 11:41 AM, Tim Lossen <tim@lossen.de> wrote:
>>
>>> hi,
>>>
>>> is it possible for a faulty client to "corrupt" a topic on the broker,
>>> so that consumers cannot consume messages any more? or does
>>> the broker protect itself against this?
>>>
>>> i am asking because we seem to have run into such a situation.
>>> we are using a perl producer and a ruby consumer. the per lib might
>>> be a bit outdated.
>>>
>>> cheers
>>> tim
>>>
>>> --
>>> http://tim.lossen.de
>>>
>>>
>
> --
> http://tim.lossen.de
>
>

Mime
View raw message