kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Fabien Chaillou (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (KAFKA-6209) Lag is inconsistent when manually committing offset for transactionnal messages
Date Tue, 14 Nov 2017 20:34:00 GMT

     [ https://issues.apache.org/jira/browse/KAFKA-6209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Fabien Chaillou updated KAFKA-6209:
-----------------------------------
    Description: 
Hi,

I'm using kafka consumers paired with kafka producers to have transactionnal consume ->
produce -> commit flow.
I checked the kafka-consumer-groups.sh tool to make sure my consumer were consuming all the
messages and it appears that my consumer always have a lag of 1.

After more digging, I discovered that kafka uses Control records for transaction handling
and those records are not returned to the consumer. So if the last record published in the
topic is part of a transaction then kafka will add a control record after it and the client
code will only be able to send a commit request up to the last control record and the lag
will be computed as 1.

It is not really a bug per se but an inconsistent behavior and I had to dig into the exactly-once
KIP (KIP-98) and the consumer code to figure this out.

I think the issue should at least be documented somewhere as I'm honestly not sure of the
proper fix. 

Thanks for your feedback.
Fabien


  was:
Hi,

I'm using kafka consumers paired with kafka producers to have transactionnal consume ->
produce -> commit flow.
I checked the kafka-consumer-groups.sh tool to make sure my consumer were consuming all the
messages and it appears that my consumer always have a lag of 1.

After more digging, I discovered that kafka uses Control records for transaction handling
and those records are not returned to the consumer. So if the last record published in the
topic is part of a transaction then kafka will add a control record after it and the client
code will only be able to send a commit request up to the last control record and the lag
will be computed as 1.

It is not really a bug per se but an inconsistent behavior and I add to dig into the exactly-once
KIP (KIP-98) and the consumer code to figure this out.

I think the issue should at least be documented somewhere as I'm honestly not sure of the
proper fix. 

Thanks for your feedback.
Fabien



> Lag is inconsistent when manually committing offset for transactionnal messages
> -------------------------------------------------------------------------------
>
>                 Key: KAFKA-6209
>                 URL: https://issues.apache.org/jira/browse/KAFKA-6209
>             Project: Kafka
>          Issue Type: Improvement
>          Components: consumer
>    Affects Versions: 0.11.0.1
>            Reporter: Fabien Chaillou
>
> Hi,
> I'm using kafka consumers paired with kafka producers to have transactionnal consume
-> produce -> commit flow.
> I checked the kafka-consumer-groups.sh tool to make sure my consumer were consuming all
the messages and it appears that my consumer always have a lag of 1.
> After more digging, I discovered that kafka uses Control records for transaction handling
and those records are not returned to the consumer. So if the last record published in the
topic is part of a transaction then kafka will add a control record after it and the client
code will only be able to send a commit request up to the last control record and the lag
will be computed as 1.
> It is not really a bug per se but an inconsistent behavior and I had to dig into the
exactly-once KIP (KIP-98) and the consumer code to figure this out.
> I think the issue should at least be documented somewhere as I'm honestly not sure of
the proper fix. 
> Thanks for your feedback.
> Fabien



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message