kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Guozhang Wang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-6150) Make Repartition Topics Transient
Date Tue, 31 Oct 2017 17:02:02 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-6150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16227114#comment-16227114

Guozhang Wang commented on KAFKA-6150:

Looked into the KIP-107 implementations, and here are a few observations:

1) currently the purgeBefore API is only implemented in the old Scala `AdminClient`, but in
the Java `KafkaAdminClient`. So we'd either a) implement the DeleteRecordsRequest in our StreamsKafkaClient
but delay its deprecation, or b) add this support in Java client, or c) use the Scala client
that introduces another dependency.

2) the DeleteRecordsRequest is a blocking request, i.e. the response would not be returned
until completion. So we'd be careful about how frequent / when to send the request.

> Make Repartition Topics Transient
> ---------------------------------
>                 Key: KAFKA-6150
>                 URL: https://issues.apache.org/jira/browse/KAFKA-6150
>             Project: Kafka
>          Issue Type: Improvement
>          Components: streams
>            Reporter: Guozhang Wang
>            Assignee: Guozhang Wang
>              Labels: operability
> Unlike changelog topics, the repartition topics could just be short-lived. Today users
have different ways to configure them with short retention such as enforce a short retention
period or use AppendTime for repartition topics. All these would be cumbersome and Streams
should just do this for the users.
> One way to do it is use the “purgeData” admin API (KIP-107) such that after the offset
of the input topics are committed, if the input topics are actually repartition topics, we
would purge the data immediately.

This message was sent by Atlassian JIRA

View raw message