kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jay Kreps (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (KAFKA-739) Handle null values in Message payload
Date Mon, 04 Mar 2013 22:57:12 GMT

     [ https://issues.apache.org/jira/browse/KAFKA-739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Jay Kreps updated KAFKA-739:

    Attachment: KAFKA-739-v1.patch

This patch is more extensive than I expected because I found a hole in the logic when handling
deletes in the log compactor. The changes are as follows:

1. Handle null properly in Message.scala and miscellaneous other places.
2. Fix the logic for handling deletes. Previously we guaranteed that we would retain delete
records only in the dirty section of the log. This is not sufficient, because a bootstrapping
consumer might see a message, but the subsequent delete message might be gc'd before the consumer
sees it.
3. OffsetMap.scala: make the map exact using a probing scheme. This means that the tail of
the log is actually now fully deduplicated. The motivation for this is making delete-handling
easier since to remove a delete tombstone you need to ensure that there are no prior occurrences
of that message. Also added a counter on the number of collisions, just to help with any debugging.
4. Added a new configuration log.cleaner.delete.retention.ms that controls the length of time
for which delete records are retained. This is implicitly a limit on the amount of time the
consumer can spend bootstrapping and still get a consistent bootstrap. Once the topic-level
config patch goes in, this will be made available at the topic level and can be set with the
create topic tool
5. Added a peek() method to iterator template. Didn't end up using it, but it is a useful
6. Changed the integration test tool to issue deletes and changed the verification to handle
delete records properly. Redid testing now with deletes included.
7. Added a variety of unit tests for null messages

> Handle null values in Message payload
> -------------------------------------
>                 Key: KAFKA-739
>                 URL: https://issues.apache.org/jira/browse/KAFKA-739
>             Project: Kafka
>          Issue Type: Bug
>            Reporter: Jay Kreps
>            Assignee: Jay Kreps
>             Fix For: 0.8.1
>         Attachments: KAFKA-739-v1.patch
> Add tests for null message payloads in producer, server, and consumer.
> Ensure log cleaner treats these as deletes.
> Test that null keys are rejected on dedupe logs.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message