kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jiangjie Qin (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-4340) Change the default value of log.message.timestamp.difference.max.ms to the same as log.retention.ms
Date Wed, 24 May 2017 20:22:04 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16023619#comment-16023619
] 

Jiangjie Qin commented on KAFKA-4340:
-------------------------------------

[~edenhill] I think one of the well established guarantee in producing is that either the
entire batch succeeds or the entire batch fails. A lot of the code are actually built on top
of this guarantee, for example the callback and offsets in the callback.

So if one of the messages in the batch had problem, it is by current design the entire batch
will fail. I do agree that in this case the producer does not really have much to do, but
that seems not a problem introduced by this patch. 

Silently discard the message on the broker side will introduce other problems because the
producer acks are on a batch. The producer will assume the message has been successfully sent
while it has been removed. And this also breaks the log retention enforcing order which always
remove an old segment before deleting messages in newer segments. 

> Change the default value of log.message.timestamp.difference.max.ms to the same as log.retention.ms
> ---------------------------------------------------------------------------------------------------
>
>                 Key: KAFKA-4340
>                 URL: https://issues.apache.org/jira/browse/KAFKA-4340
>             Project: Kafka
>          Issue Type: Improvement
>          Components: core
>    Affects Versions: 0.10.1.0
>            Reporter: Jiangjie Qin
>            Assignee: Jiangjie Qin
>             Fix For: 0.11.0.0
>
>
> [~junrao] brought up the following scenario: 
> If users are pumping data with timestamp already passed log.retention.ms into Kafka,
the messages will be appended to the log but will be immediately rolled out by log retention
thread when it kicks in and the messages will be deleted. 
> To avoid this produce-and-deleted scenario, we can set the default value of log.message.timestamp.difference.max.ms
to be the same as log.retention.ms.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message