kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Magnus Edenhill (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-4340) Change the default value of log.message.timestamp.difference.max.ms to the same as log.retention.ms
Date Wed, 24 May 2017 20:52:04 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16023668#comment-16023668

Magnus Edenhill commented on KAFKA-4340:

Generally I would agree, but in this case I don't think there is a practical difference to
discarding outdated messages, or removing them with the retention cleaner, it is just a matter
of time frame - discarding and not appending outdated messages to the logs is immediate, while
the retention cleaner might kick in immediately or in whatever max interval it is configured
So from the producer and consumer perspectives the end result is pretty much the same: there
is no guarantee that an outdated message will be seen by a consumer.

However, rejecting the entire batch means there will be guaranteed data loss in the message
stream: the producer will not try to re-send those failed messages, even if all but one message
in the batch were actually okay. I strongly feel this is undesired behaviour from the application's
point of view.

> Change the default value of log.message.timestamp.difference.max.ms to the same as log.retention.ms
> ---------------------------------------------------------------------------------------------------
>                 Key: KAFKA-4340
>                 URL: https://issues.apache.org/jira/browse/KAFKA-4340
>             Project: Kafka
>          Issue Type: Improvement
>          Components: core
>    Affects Versions:
>            Reporter: Jiangjie Qin
>            Assignee: Jiangjie Qin
>             Fix For:
> [~junrao] brought up the following scenario: 
> If users are pumping data with timestamp already passed log.retention.ms into Kafka,
the messages will be appended to the log but will be immediately rolled out by log retention
thread when it kicks in and the messages will be deleted. 
> To avoid this produce-and-deleted scenario, we can set the default value of log.message.timestamp.difference.max.ms
to be the same as log.retention.ms.

This message was sent by Atlassian JIRA

View raw message