storm-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kumar Gaurav (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (STORM-418) using storm kafka spout doesn't scale when increasing topic partition
Date Thu, 24 Jul 2014 06:18:40 GMT

    [ https://issues.apache.org/jira/browse/STORM-418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14072873#comment-14072873
] 

Kumar Gaurav commented on STORM-418:
------------------------------------

Hi Varun,

yes, we increased spout parallelism along with kafka topic partition increase. The message
size we are using for this testing was 80-85 bytes.

Let me know if you need any more details.

Thanks
Gaurav


> using storm kafka spout doesn't scale when increasing topic partition
> ---------------------------------------------------------------------
>
>                 Key: STORM-418
>                 URL: https://issues.apache.org/jira/browse/STORM-418
>             Project: Apache Storm (Incubating)
>          Issue Type: Bug
>    Affects Versions: 0.9.2-incubating
>         Environment: EC2 c3.8x large instances for supervisor and kafka-0.8.1.1 running
Amazon Linux 64 bit version OS.
> Kafka is using 4 EBS drives of 4000 IOPS of size 150 GB each.
>            Reporter: Kumar Gaurav
>              Labels: performance
>
> We are running a topology of kafka spout + single bolt on a single supervisor machine.
> Bolt functionality is limited to log parsed data from spout only.
> When Trying to consume same type of data by different topics using varying topic partitions
topology throughput remain same in order of 70000 msg/sec.
> However, when multiple spout instances are used in same topology , we were able to increase
topology throughput to more than 100,000 msg/sec.
>  We have tried to increase kafka topic partition from 8 to 200, however topology throughput
remains same for sinlge spout instance.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message