kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Luca Toscano (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-1379) Partition reassignment resets clock for time-based retention
Date Wed, 01 Jun 2016 10:08:59 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-1379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15310068#comment-15310068
] 

Luca Toscano commented on KAFKA-1379:
-------------------------------------

Hi Moritz,

thanks a lot for pointing us to this Jira in users@. At the moment we use a similar trick
to resolve disk partitions filling up (retention.ms):
https://wikitech.wikimedia.org/wiki/Analytics/Cluster/Kafka/Administration#Temporarily_Modify_Per_Topic_Retention_Settings

I also opened a Phabricator task to track this problem https://phabricator.wikimedia.org/T136690

retention.bytes is definitely worth to try, but is there anything else that can mitigate this
issue?

> Partition reassignment resets clock for time-based retention
> ------------------------------------------------------------
>
>                 Key: KAFKA-1379
>                 URL: https://issues.apache.org/jira/browse/KAFKA-1379
>             Project: Kafka
>          Issue Type: Bug
>            Reporter: Joel Koshy
>
> Since retention is driven off mod-times reassigned partitions will result in
> data that has been on a leader to be retained for another full retention
> cycle. E.g., if retention is seven days and you reassign partitions on the
> sixth day then those partitions will remain on the replicas for another
> seven days.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message