storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "唐思成" <jadetan...@qq.com>
Subject Re: Re: Max Spout Pending
Date Tue, 15 Jul 2014 04:26:39 GMT
https://gist.github.com/mrflip/5958028#provisionings 
Max-pending (TOPOLOGY_MAX_SPOUT_PENDING) sets the number of tuple trees live in the system
at any one time.
maybe this is useful for you



2014-07-15 



唐思成 



发件人: Raphael Hsieh 
发送时间: 2014-07-15  05:54:28 
收件人: user 
抄送: 
主题: Re: Max Spout Pending 
 
Is there a way to tell how many batches my topology processes per second ?
Or for that matter how many tuples are processed per second ?
Aside from creating a new bolt purely for that aggregation ?




On Mon, Jul 14, 2014 at 2:08 PM, Carlos Rodriguez <crodriguez@redborder.net> wrote:

Max spout pending config specifies how many *batches* can be processed simultaneously by your
topology.
Thats why 48,000 seems absurdly high to you. Divide it between the batch size and you'll get
the max spout pending config that you were expecting.




2014-07-14 19:00 GMT+02:00 Raphael Hsieh <raffihsieh@gmail.com>:


What is the optimal max spout pending to use in a topology ? 
I found this thread here: http://mail-archives.apache.org/mod_mbox/storm-user/201402.mbox/%3CCA+AvHzatFG_s88LkOMbVOMmKH-rAfWr6Szy0i8B8Tm3RFAB-7g@mail.gmail.com%3E
that didn't seem to have a follow up. 

Part of it says to

"Start with a max spout pending that is for sure too small -- one for
trident, or the number of executors for storm -- and increase it until you
stop seeing changes in the flow. You'll probably end up with something
near 2*(throughput
in recs/sec)*(end-to-end latency) (2x the Little's law capacity)."
Does this make sense for a Max Spout Pending value ?
I expect my topology to have a throughput of around 80,000/s and I've been seeing a complete
latency of around 300ms, so given this formula, I'd want 2*80000*.3 = 48,000 Max Spout Pending.

This seems absurdly high to me..


-- 

Raphael Hsieh

 
 






-- 

Carlos Rodríguez
Developer at ENEO Tecnología
http://redborder.net/

http://lnkd.in/bgfCVF9






-- 

Raphael Hsieh

 
 
Mime
View raw message