kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Apurva Mehta (JIRA)" <j...@apache.org>
Subject [jira] [Created] (KAFKA-5865) Expiring batches with idempotence enabled could cause data loss.
Date Sat, 09 Sep 2017 00:48:01 GMT
Apurva Mehta created KAFKA-5865:
-----------------------------------

             Summary: Expiring batches with idempotence enabled could cause data loss.
                 Key: KAFKA-5865
                 URL: https://issues.apache.org/jira/browse/KAFKA-5865
             Project: Kafka
          Issue Type: Bug
    Affects Versions: 0.11.0.0
            Reporter: Apurva Mehta
            Assignee: Apurva Mehta
             Fix For: 1.0.0


Currently we have a problem with this line:
https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java#L282

Because we can reset the producer id and return after draining batches, it means that we can
drain batches for some partitions, then find a batch has expired, and then return. But the
batches which were drained are now no longer in the producer queue, and haven't been sent.
Thus they are totally lost, and the call backs will never be invoked.

This is already fixed in https://github.com/apache/kafka/pull/3743 , but opening this in case
we want to fix it in 0.11.0.2 as well.




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message