kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Manikumar (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (KAFKA-4951) KafkaProducer may send duplicated message sometimes
Date Wed, 16 Aug 2017 17:13:00 GMT

     [ https://issues.apache.org/jira/browse/KAFKA-4951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Manikumar resolved KAFKA-4951.
------------------------------
    Resolution: Fixed

This scenario is handled in the Idempotent producer (KIP-98) released in Kafka 0.11.0.0. 
Pl reopen if you think the issue still exists

> KafkaProducer may send duplicated message sometimes
> ---------------------------------------------------
>
>                 Key: KAFKA-4951
>                 URL: https://issues.apache.org/jira/browse/KAFKA-4951
>             Project: Kafka
>          Issue Type: Bug
>    Affects Versions: 0.9.0.1
>            Reporter: cuiyang
>
> I foud that KafkaProducer may send duplicated message sometimes, which is happend when:
>      In Sender thread:
>      NetworkClient::poll()
>          -> this.selector.poll()    //send the message, such as "abc", and send it
to broker successfully
>          -> handleTimedOutRequests(responses,updatedNow);  //Judge whether the message
 "abc" which is sent above is expired or timeout,  and the judge is  based on the parameter
 this.requestTimeoutMs and updatedNow;  
>          -> response.request().callback().onComplete()
>              -> completeBatch(batch,Errors.NETWORK_EXCEPTION,-1L,correlationId,now);
  //If themessage was judged as expired, then it will be reenqueued and send repeatly next
loop;
>                  -> this.accumulator.reenqueue(batch,now);
> The problem comes out:  If the message "abc" is sent successfully to broker, but it may
be judged to expired, so the message will be sent repeately next loop, which make the message
duplicated.
> I can reproduce this scenario normally.
> In my opinion, I think Send::handleTimedOutRequests() is not much useful, because the
response of sending request from broker is succesfully and has no error, which means brokers
persist it successfully. And this function  will induce to the duplicated message problems.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message