kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun Rao <...@confluent.io>
Subject Re: [VOTE] KIP-280: Enhanced log compaction
Date Tue, 03 Jul 2018 21:58:42 GMT
Hi, Luis,

Thanks for the KIP. Overall, this seems a useful KIP. A few comments below.

1. I guess both new configurations will be at the topic level?
2. Since the log cleaner now needs to keep both the offset and another long
(say timestamp) in the de-dup map, it reduces the number of keys that we
can keep in the map and thus may require more rounds of cleaning. This is
probably not a big issue, but it will be useful to document this impact in
the KIP.
3. With the new cleaning strategy, we want to be a bit careful with
removing the last message in a partition (which is possible now). We need
to preserve the offset of the last message so that we don't reuse the
offset for a different message. One way to simply never remove the last
message. Another way (suggested by Jason) is to create an empty message
batch.

Jun

On Sat, Jun 9, 2018 at 12:39 AM, Luís Cabral <Luis_Cabral@yahoo.com.invalid>
wrote:

> Hi all,
>
> Any takers on having a look at this KIP and voting on it?
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 280%3A+Enhanced+log+compaction
>
> Cheers,
> Luis
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message