Mailing-List: contact jira-help@kafka.apache.org; run by ezmlm
Precedence: bulk
Reply-To: jira@kafka.apache.org
Date: Tue, 14 Nov 2017 20:34:00 +0000 (UTC)
From: "Fabien Chaillou (JIRA)" <jira@apache.org>
To: jira@kafka.apache.org
Message-ID: <JIRA.13118345.1510689074000.237874.1510691640072@Atlassian.JIRA>
In-Reply-To: <JIRA.13118345.1510689074000@Atlassian.JIRA>
References: <JIRA.13118345.1510689074000@Atlassian.JIRA> <JIRA.13118345.1510689074668@jira-lw-us.apache.org>
Subject: [jira] [Updated] (KAFKA-6209) Lag is inconsistent when manually
 committing offset for transactionnal messages
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Tue, 14 Nov 2017 20:34:04 -0000


     [ https://issues.apache.org/jira/browse/KAFKA-6209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Fabien Chaillou updated KAFKA-6209:
-----------------------------------
    Description: 
Hi,

I'm using kafka consumers paired with kafka producers to have transactionnal consume -> produce -> commit flow.
I checked the kafka-consumer-groups.sh tool to make sure my consumer were consuming all the messages and it appears that my consumer always have a lag of 1.

After more digging, I discovered that kafka uses Control records for transaction handling and those records are not returned to the consumer. So if the last record published in the topic is part of a transaction then kafka will add a control record after it and the client code will only be able to send a commit request up to the last control record and the lag will be computed as 1.

It is not really a bug per se but an inconsistent behavior and I had to dig into the exactly-once KIP (KIP-98) and the consumer code to figure this out.

I think the issue should at least be documented somewhere as I'm honestly not sure of the proper fix. 

Thanks for your feedback.
Fabien


  was:
Hi,

I'm using kafka consumers paired with kafka producers to have transactionnal consume -> produce -> commit flow.
I checked the kafka-consumer-groups.sh tool to make sure my consumer were consuming all the messages and it appears that my consumer always have a lag of 1.

After more digging, I discovered that kafka uses Control records for transaction handling and those records are not returned to the consumer. So if the last record published in the topic is part of a transaction then kafka will add a control record after it and the client code will only be able to send a commit request up to the last control record and the lag will be computed as 1.

It is not really a bug per se but an inconsistent behavior and I add to dig into the exactly-once KIP (KIP-98) and the consumer code to figure this out.

I think the issue should at least be documented somewhere as I'm honestly not sure of the proper fix. 

Thanks for your feedback.
Fabien


> Lag is inconsistent when manually committing offset for transactionnal messages
> -------------------------------------------------------------------------------
>
>                 Key: KAFKA-6209
>                 URL: https://issues.apache.org/jira/browse/KAFKA-6209
>             Project: Kafka
>          Issue Type: Improvement
>          Components: consumer
>    Affects Versions: 0.11.0.1
>            Reporter: Fabien Chaillou
>
> Hi,
> I'm using kafka consumers paired with kafka producers to have transactionnal consume -> produce -> commit flow.
> I checked the kafka-consumer-groups.sh tool to make sure my consumer were consuming all the messages and it appears that my consumer always have a lag of 1.
> After more digging, I discovered that kafka uses Control records for transaction handling and those records are not returned to the consumer. So if the last record published in the topic is part of a transaction then kafka will add a control record after it and the client code will only be able to send a commit request up to the last control record and the lag will be computed as 1.
> It is not really a bug per se but an inconsistent behavior and I had to dig into the exactly-once KIP (KIP-98) and the consumer code to figure this out.
> I think the issue should at least be documented somewhere as I'm honestly not sure of the proper fix. 
> Thanks for your feedback.
> Fabien


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)