storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Filipa Moura <filipa.mendesmo...@gmail.com>
Subject Re: kafkaspout is very slow
Date Wed, 04 Feb 2015 22:24:52 GMT
can you share a  screenshot of the Storm UI for your spout?

On Wed, Feb 4, 2015 at 9:58 PM, clay teahouse <clayteahouse@gmail.com>
wrote:

>  I have this issue with any amount of load. Different max spout pendings
> do not seem to make much a difference. I've lowered this parameter to 100,
> still a little difference . At this point the bolt consuming the data does
> no processing.
>
> On Wed, Feb 4, 2015 at 3:26 PM, Haralds Ulmanis <haralds@evilezh.net>
> wrote:
>
>> I'm not sure, that i understand your problem .. but here is few points:
>> If you have large pending spout size and slow processing - you will see
>> large latency at kafka spout probably. Spout emits message .. it stays in
>> queue for long time (that will add latency) .. and finally is processed and
>> ack received. You will see queue time + processing time in kafka spout
>> latency.
>> Take a look at load factors of your bolts - are they close to 1 or more ?
>> and load factor of kafka spout.
>>
>> On 4 February 2015 at 21:19, Andrey Yegorov <andrey.yegorov@gmail.com>
>> wrote:
>>
>>> have you tried increasing max spout pending parameter for the spout?
>>>
>>> builder.setSpout("kafka",
>>>                        new KafkaSpout(spoutConfig),
>>>                        TOPOLOGY_NUM_TASKS_KAFKA_SPOUT)
>>>           .setNumTasks(TOPOLOGY_NUM_TASKS_KAFKA_SPOUT)
>>>           //the maximum parallelism you can have on a KafkaSpout is the
>>> number of partitions
>>>           .setMaxSpoutPending(*TOPOLOGY_MAX_SPOUT_PENDING*);
>>>
>>> ----------
>>> Andrey Yegorov
>>>
>>> On Tue, Feb 3, 2015 at 4:03 AM, clay teahouse <clayteahouse@gmail.com>
>>> wrote:
>>>
>>>> Hi all,
>>>>
>>>> In my topology,  kafka spout is responsible for over 85% of the
>>>> latency. I have tried different spout max pending and played with the
>>>> buffer size and fetch size, still no luck. Any hint on how to optimize the
>>>> spout? The issue doesn't seem to be with the kafka side, as I see high
>>>> throughput with the simple kafka consumer.
>>>>
>>>> thank you for your feedback
>>>> Clay
>>>>
>>>>
>>>
>>
>

Mime
View raw message