kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jun Rao (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-4340) Change the default value of log.message.timestamp.difference.max.ms to the same as log.retention.ms
Date Wed, 24 May 2017 23:16:04 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16023862#comment-16023862

Jun Rao commented on KAFKA-4340:

[~edenhill], as Jiangjie said, currently, the broker takes the whole batch as the unit. So,
it's a bit weird for the broker to take only some of the messages in a batch. This is especially
true with the EOS work in which the producer wants to commit multiple messages all or nothing.
If the broker silently rejects some messages without telling the producer, the EOS guarantees
will be broken.

Your point is valid though in that if the broker rejects the whole batch, currently the client
doesn't know which messages fail the timestamp check. One way to address this is to include
the array index of messages failing the timestamp check in the partition level produce response.
Then, the client still fails all messages in the batch but with different error code in the
callback, depending on whether the message fails the timestamp check or not. That way, the
application can choose to resend failed messages with good timestamp, if desired. 

> Change the default value of log.message.timestamp.difference.max.ms to the same as log.retention.ms
> ---------------------------------------------------------------------------------------------------
>                 Key: KAFKA-4340
>                 URL: https://issues.apache.org/jira/browse/KAFKA-4340
>             Project: Kafka
>          Issue Type: Improvement
>          Components: core
>    Affects Versions:
>            Reporter: Jiangjie Qin
>            Assignee: Jiangjie Qin
>             Fix For:
> [~junrao] brought up the following scenario: 
> If users are pumping data with timestamp already passed log.retention.ms into Kafka,
the messages will be appended to the log but will be immediately rolled out by log retention
thread when it kicks in and the messages will be deleted. 
> To avoid this produce-and-deleted scenario, we can set the default value of log.message.timestamp.difference.max.ms
to be the same as log.retention.ms.

This message was sent by Atlassian JIRA

View raw message