incubator-kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jay Kreps (JIRA)" <>
Subject [jira] [Created] (KAFKA-631) Implement log compaction
Date Mon, 26 Nov 2012 22:30:58 GMT
Jay Kreps created KAFKA-631:

             Summary: Implement log compaction
                 Key: KAFKA-631
             Project: Kafka
          Issue Type: New Feature
          Components: core
    Affects Versions: 0.8.1
            Reporter: Jay Kreps

Currently Kafka has only one way to bound the space of the log, namely by deleting old segments.
The policy that controls which segments are deleted can be configured based either on the
number of bytes to retain or the age of the messages. This makes sense for event or log data
which has no notion of primary key. However lots of data has a primary key and consists of
updates by primary key. For this data it would be nice to be able to ensure that the log contained
at least the last version of every key.

As an example, say that the Kafka topic contains a sequence of User Account messages, each
capturing the current state of a given user account. Rather than simply discarding old segments,
since the set of user accounts is finite, it might make more sense to delete individual records
that have been made obsolete by a more recent update for the same key. This would ensure that
the topic contained at least the current state of each record.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see:

View raw message