kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ewen Cheslack-Postava <e...@confluent.io>
Subject Re: Kafka New Consumer Performance Test ThroughPut Degradation
Date Thu, 27 Aug 2015 23:21:27 GMT
I don't think the commands are really equivalent despite just adding the
--new-consumer flag. ConsumerPerformance uses a single thread when using
the new consumer (it literally just allocates the consumer, loops until
it's consumed enough, then exits), whereas the old consumer uses a bunch of
additional threads.

To really compare performance, someone would have to think through a fair
way to compare them -- the two operate so differently that you'd have to be
very careful to get an apples-to-apples comparison.

By the way, membership in consumer groups should be a lot cheaper with the
new consumer (the ZK coordination issues with lots of consumers aren't a
problem since ZK is not involved), so you can probably scale up the number
of consumer threads with little impact. It might be nice to patch the
consumer perf test to respect the # of threads setting, which might be a
first step to getting a more reasonable comparison.

-Ewen

On Thu, Aug 27, 2015 at 11:25 AM, Poorna Chandra Tejashvi Reddy <
pctreddy@gmail.com> wrote:

> Hi,
>
> We have built the latest kafka from https://github.com/apache/kafka based
> on this commit id 436b7ddc386eb688ba0f12836710f5e4bcaa06c8 .
> We ran the performance test on a 3 node kafka cluster. There is a huge
> throughput degradation using the new-consumer compared to the regular
> consumer. Below are the numbers that explain the same.
>
> bin/kafka-consumer-perf-test.sh --zookeeper zkIp:2181 --broker-list
> brokerIp:9092 --topics test --messages 5000000 : gives a throughput of 693
> K
>
> bin/kafka-consumer-perf-test.sh --zookeeper zkIp:2181 --broker-list
> brokerIp:9092 --topics test --messages 5000000 --new-consumer : gives a
> throughput of  51k
>
> The whole set up is based on ec2, Kafka brokers running on r3.2x large.
>
> Are you guys aware of this performance degradation , do you have a JIRA for
> this, which can be used to track the resolution.
>
>
> Thanks,
>
> -Poorna
>



-- 
Thanks,
Ewen

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message