Yes that is basically right. If indeed your executors are processing enough data to be utilizing
nearly all of cpu then you should consider 1 executor per core. May be a good idea to budget
1020% cpu per machine for other stuff including interactive login.
Sent from Yahoo Mail for iPhone
On Tuesday, April 9, 2019, 2:25 PM, Jayant Sharma <sharmajayant27@gmail.com> wrote:
Hi,
I was going through P. Taylor Goetz's talk on scaling storm cluster (2014). He made an interesting
point of computing available parallelism units in a cluster. For CPU bound applications this
limit was 1 executor/CPU core. I just wanted to be sure my understanding about this is clear:
If I have 3 supervisor machines, each having 5 workers or JVMs and each machine has 16 CPU
cores. I have 3*16 = 48 parallelism units to distribute among all of my topologies? Which
means the sum of all spouts and bolts executors across all the topologies should be 48. What
are the implications if I keep my executors more or less than this value?
If my understanding is incorrect, can someone please explain how to compute parallelism units
and relate them to number of executors.
Thanks,Jayant Sharma
