kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ewen Cheslack-Postava <e...@confluent.io>
Subject Re: [DISCUSS] KIP-107: Add purgeDataBefore() API in AdminClient
Date Tue, 03 Jan 2017 23:56:30 GMT
Dong,

Looks like that's an internal link,
https://cwiki.apache.org/confluence/display/KAFKA/KIP-107%3A+Add+purgeDataBefore%28%29+API+in+AdminClient
is the right one.

I have a question about one of the rejected alternatives:

> Using committed offset instead of an extra API to trigger data purge
operation.

The KIP says this would be more complicated to implement. Why is that? I
think brokers would have to consume the entire offsets topic, but the data
stored in memory doesn't seem to change and applying this when updated
offsets are seen seems basically the same. It might also be possible to
make it work even with multiple consumer groups if that was desired
(although that'd require tracking more data in memory) as a generalization
without requiring coordination between the consumer groups. Given the
motivation, I'm assuming this was considered unnecessary since this
specifically targets intermediate stream processing topics.

Another question is why expose this via AdminClient (which isn't public API
afaik)? Why not, for example, expose it on the Consumer, which is
presumably where you'd want access to it since the functionality depends on
the consumer actually having consumed the data?

-Ewen

On Tue, Jan 3, 2017 at 2:45 PM, Dong Lin <lindong28@gmail.com> wrote:

> Hi all,
>
> We created KIP-107 to propose addition of purgeDataBefore() API in
> AdminClient.
>
> Please find the KIP wiki in the link https://iwww.corp.linkedin.
> com/wiki/cf/display/ENGS/Kafka+purgeDataBefore%28%29+API+design+proposal.
> We
> would love to hear your comments and suggestions.
>
> Thanks,
> Dong
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message