kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jiangjie Qin (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-4099) Change the time based log rolling to only based on the message timestamp.
Date Tue, 25 Oct 2016 01:40:58 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-4099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15603857#comment-15603857
] 

Jiangjie Qin commented on KAFKA-4099:
-------------------------------------

[~junrao] Good point about making log.message.timestamp.difference.max.ms the same as log.retention.ms.
That makes sense. This change may have some impact on the users who is currently pumping data
into Kafka and consume it immediately before the log retention thread kicks in. But it should
be rare and seems fragile anyway.

> Change the time based log rolling to only based on the message timestamp.
> -------------------------------------------------------------------------
>
>                 Key: KAFKA-4099
>                 URL: https://issues.apache.org/jira/browse/KAFKA-4099
>             Project: Kafka
>          Issue Type: Bug
>          Components: core
>            Reporter: Jiangjie Qin
>            Assignee: Jiangjie Qin
>             Fix For: 0.10.1.0
>
>
> This is an issue introduced in KAFKA-3163. When partition relocation occurs, the newly
created replica may have messages with old timestamp and cause the log segment rolling for
each message. The fix is to change the log rolling behavior to only based on the message timestamp
when the messages are in message format 0.10.0 or above. If the first message in the segment
does not have a timetamp, we will fall back to use the wall clock time for log rolling.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message