kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stanislav Kozlovski (JIRA)" <j...@apache.org>
Subject [jira] [Created] (KAFKA-7910) Document retention.ms behavior with record timestamp
Date Fri, 08 Feb 2019 12:54:00 GMT
Stanislav Kozlovski created KAFKA-7910:

             Summary: Document retention.ms behavior with record timestamp
                 Key: KAFKA-7910
                 URL: https://issues.apache.org/jira/browse/KAFKA-7910
             Project: Kafka
          Issue Type: Improvement
            Reporter: Stanislav Kozlovski

It is intuitive to believe that `log.retention.ms` starts applying once a log file is closed.
The documentation says:
> This configuration controls the maximum time we will retain a log before we will discard
old log segments to free up space if we are using the "delete" retention policy.
Yet, the actual behavior is that we take into account the largest timestamp of that segment
file ([https://github.com/apache/kafka/blob/4cdbb3e5c19142d118f0f3999dd3e21deccb3643/core/src/main/scala/kafka/log/Log.scala#L1246)|https://github.com/apache/kafka/blob/4cdbb3e5c19142d118f0f3999dd3e21deccb3643/core/src/main/scala/kafka/log/Log.scala#L1246).] and
then consider `retention.ms` on top of that.

This means that if Kafka is configured with `log.message.timestamp.type=CreateTime` (as it
is by default), any records that have a future timestamp set by the producer will not get
deleted as expected by the initial intuition (and documentation) of the `log.retention.ms`.

We should document the behavior of `retention.ms` with the record timestamp.

This message was sent by Atlassian JIRA

View raw message