kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Damian Guy <damian....@gmail.com>
Subject Re: [DISCUSS] KIP-71 Enable log compaction and deletion to co-exist
Date Fri, 12 Aug 2016 13:39:45 GMT
Hi Jason,

It is still not clear to me how adding a callback would help. You could
remove the periodic task, but you would still need to scan the entire cache
to remove the expired offsets. Am i missing something?

Thanks,
Damian


On 11 August 2016 at 10:00, Damian Guy <damian.guy@gmail.com> wrote:

> Hi Jason,
>
> Thanks for your input - appreciated.
>
> 1. Would it make sense to use this KIP in the consumer coordinator to
>> expire offsets based on the topic's retention time? Currently, we have a
>> periodic task which scans the full cache to check which offsets can be
>> expired, but we might be able to get rid of this if we had a callback to
>> update the cache when a segment was deleted. Technically offsets can be
>> given their own expiration time, but it seems questionable whether we need
>> this going forward (the new consumer doesn't even expose it at the
>> moment).
>>
>
> The KIP in its current form isn't adding a callback. So you'd still need
> to scan the cache and remove any expired offsets, however you wouldn't send
> the tombstone messages.
> Having a callback sounds useful, though it isn't clear to me how you would
> know which offsets to remove from the cache on segment deletion? I will
> look into it.
>
>
>> 2. This KIP could also be useful for expiration in the case of a cache
>> maintained on the client, but I don't see an obvious way that we'd be able
>> to leverage it since there's no indication to the client when a segment
>> has
>> been deleted (unless they reload the cache from the beginning of the log).
>> One approach I can think of would be to write corresponding tombstones as
>> necessary when a segment is removed, but that seems pretty heavy. Have you
>> considered this problem?
>>
>>
> We've not considered this and I'm not sure we want to as part of this KIP.
>
> Thanks,
> Damian
>
>
>>
>>
>> On Mon, Aug 8, 2016 at 12:41 AM, Damian Guy <damian.guy@gmail.com> wrote:
>>
>> > Hi,
>> >
>> > We have created KIP 71: Enable log compaction and deletion to co-exist`
>> >
>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>> > 71%3A+Enable+log+compaction+and+deletion+to+co-exist
>> >
>> > Please take a look. Feedback is appreciated.
>> >
>> > Thank you
>> >
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message