storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Danijel Schiavuzzi <dani...@schiavuzzi.com>
Subject Optimizing Trident -- optimal max spout pending value?
Date Wed, 05 Feb 2014 19:26:34 GMT
Hi,

I'm working with Trident and building a Trident Transactional topology
using the https://github.com/wurstmeister/storm-kafka-0.8-plus/ Kafka spout.

I was wondering what is the best approach to calculating the
Config.maxSpoutPending() value for Trident? Quoting the Storm FAQ:

"Start with a max spout pending that is for sure too small -- one for
trident, or the number of executors for storm -- and increase it until you
stop seeing changes in the flow. You'll probably end up with something
near 2*(throughput
in recs/sec)*(end-to-end latency) (2x the Little's law capacity)."

My topology has an average throughput of 2000 tuples/s and an end-to-end
latency of 50 ms. Given that formula, and assuming the latency is specified
in milliseconds, my max spout pending value should be 2 * 2000 tuples/s *
0.050 s = 200 tuples.

Is that formula correct? How do I best verify if the value is optimal for
the topology? Also, what approach to setting the "Trident emit batch
interval millis" value would you recommend?

Thanks,

-- 
Danijel Schiavuzzi

E: danijel@schiavuzzi.com
W: www.schiavuzzi.com
T: +385989035562
Skype: danijels7

Mime
View raw message