flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shinhyung Yang <shinhyung.y...@gmail.com>
Subject Re: scaling a flink streaming application on a single node
Date Thu, 31 Mar 2016 17:16:19 GMT
Thank you for replying!

I am trying to do this on a single machine in fact. Since it has 64
cores, it would be interesting to look at the performance in that
regard.

> How many machines are you using for this?
>
> The fact that you are giving 64 slots to each TaskManager means that a
> single TaskManager may end up executing all 64 pipelines. That would heavily
> overload that TaskManager and cause heavy degradation.

Does it make sense if I run multiple TaskManagers on a single machine
if 64 slots are too many for a TaskManager?

> If, for example, you use 16 machines, then give each machine 4 task slots
> (total of 64 slots across all machines)
> That way, the final run (parallelism 64) will be guaranteed to be spread
> across all machines.

My intention for the experiment at the moment is to try to scale the
application up on a single machine to its maximum before moving on to
run the experiment on multiple machines.

Thank you again!
With best regards,
Shinhyung Yang

Mime
View raw message