Mailing-List: contact jira-help@kafka.apache.org; run by ezmlm
Precedence: bulk
Reply-To: jira@kafka.apache.org
Date: Thu, 17 Aug 2017 04:20:00 +0000 (UTC)
From: "Apurva Mehta (JIRA)" <jira@apache.org>
To: jira@kafka.apache.org
Message-ID: <JIRA.13078120.1496869762000.50842.1502943600529@Atlassian.JIRA>
In-Reply-To: <JIRA.13078120.1496869762000@Atlassian.JIRA>
References: <JIRA.13078120.1496869762000@Atlassian.JIRA> <JIRA.13078120.1496869762420@jira-lw-us.apache.org>
Subject: [jira] [Comment Edited] (KAFKA-5403) Transactions system test
 should dedup consumed messages by offset
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Thu, 17 Aug 2017 04:20:06 -0000


    [ https://issues.apache.org/jira/browse/KAFKA-5403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16129866#comment-16129866 ] 

Apurva Mehta edited comment on KAFKA-5403 at 8/17/17 4:19 AM:
--------------------------------------------------------------

I think we should punt on this. The problems with the patch are not easy to fix as the {{VerifiableConsumer}} validates that offsets are sequential, which is not true when you have transactions. And the {{ConsoleConsumer}} doesn't expose offsets. So modifying either without breaking compatibility will take time.

Also, the system test has been running reliably for months without suffering any problems with duplicate reads on the same offset.


was (Author: apurva):
I think we should punt on this. The problems with the patch are not easy to fix as the {{VerifiableConsumer}} validates that offsets are sequential, which is not true when you have transactions. And the {{ConsoleConsumer}} doesn't expose offsets. So modifying either breaking compatibility will take time.

Also, the system test has been running reliably for months without suffering any problems with duplicate reads on the same offset.

> Transactions system test should dedup consumed messages by offset
> -----------------------------------------------------------------
>
>                 Key: KAFKA-5403
>                 URL: https://issues.apache.org/jira/browse/KAFKA-5403
>             Project: Kafka
>          Issue Type: Bug
>    Affects Versions: 0.11.0.0
>            Reporter: Apurva Mehta
>            Assignee: Apurva Mehta
>             Fix For: 1.0.0
>
>
> In KAFKA-5396, we saw that the consumers which verify the data in multiple topics could read the same offsets multiple times, for instance when a rebalance happens. 
> This would detect spurious duplicates, causing the test to fail. We should dedup the consumed messages by offset and only fail the test if we have duplicate values for a if for a unique set of offsets.


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)