Mailing-List: contact dev-help@kafka.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@kafka.apache.org
Date: Fri, 28 Aug 2015 17:02:46 +0000 (UTC)
From: "Jay Kreps (JIRA)" <jira@apache.org>
To: dev@kafka.apache.org
Message-ID: <JIRA.12859820.1440743390000.196116.1440781366307@Atlassian.JIRA>
In-Reply-To: <JIRA.12859820.1440743390000@Atlassian.JIRA>
References: <JIRA.12859820.1440743390000@Atlassian.JIRA>
 <JIRA.12859820.1440743390447@arcas>
Subject: [jira] [Commented] (KAFKA-2486) New consumer performance
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/KAFKA-2486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14720237#comment-14720237 ] 

Jay Kreps commented on KAFKA-2486:
----------------------------------

[~guozhang] I interpret you to be saying that even if there is data, sometimes fetching immediately isn't ideal for throughput because of the reduction in batching? I think that is right but that is controlled by the min bytes setting in the fetch request, right? In other words I think we already address that. Not sure what the ideal consumer perf settings are. I think for that perf test it shoudn't matter too much what the max wait is because there is always data to fetch so it never really get's used.

I agree that right now letting the consumer set the max wait to 0 does allow the possibility of abuse but I think the fix for that is probably an eventual server-side quota.

> New consumer performance
> ------------------------
>
>                 Key: KAFKA-2486
>                 URL: https://issues.apache.org/jira/browse/KAFKA-2486
>             Project: Kafka
>          Issue Type: Sub-task
>          Components: consumer
>            Reporter: Ewen Cheslack-Postava
>            Assignee: Jason Gustafson
>             Fix For: 0.8.3
>
>
> The new consumer was previously reaching getting good performance. However, a recent report on the mailing list indicates it's dropped significantly. After evaluation, even with a local broker it seems to only be reaching a 2-10MB/s, compared to 600+MB/s previously. Before release, we should get the performance back on par.
> Some details about where the regression occurred from the mailing list http://mail-archives.apache.org/mod_mbox/kafka-dev/201508.mbox/%3CCAAdKFaE8bPSeWZf%2BF9RuA-xZazRpBrZG6vo454QLVHBAk_VOJg%40mail.gmail.com%3E :
> bq. At 49026f11781181c38e9d5edb634be9d27245c961 (May 14th), we went from good performance -> an error due to broker apparently not accepting the partition assignment strategy. Since this commit seems to add heartbeats and the server side code for partition assignment strategies, I assume we were missing something on the client side and by filling in the server side, things stopped working.
> bq. On either 84636272422b6379d57d4c5ef68b156edc1c67f8 or a5b11886df8c7aad0548efd2c7c3dbc579232f03 (July 17th), I am able to run the perf test again, but it's slow -- ~10MB/s for me vs the 2MB/s Jay was seeing, but that's still far less than the 600MB/s I saw on the earlier commits.
> Ideally we would also at least have a system test in place for the new consumer, even if regressions weren't automatically detected. It would at least allow for manually checking for regressions. This should not be difficult since there are already old consumer performance tests.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)