flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ufuk Celebi <...@apache.org>
Subject Re: scaling question
Date Fri, 19 Jun 2015 14:40:48 GMT
Hey Bill!

On 19 Jun 2015, at 16:24, Bill Sparks <jsparks@cray.com> wrote:

> Sorry for the post again. I guess I'm not understanding this… 

Thanks for posting again, not sorry! ;-) 

Regarding the broken link: where did you get this link? I think it should be referring here:
http://ci.apache.org/projects/flink/flink-docs-master/apis/programming_guide.html#parallel-execution

For smaller machines, we usually suggest to set the number of slots per task manager to the
number of cores, in your case 64. You will then have #taskmanager * #slotsPerTaskManager slots
in your cluster, e.g. 1024. This should also be the parallelism you set for your program.

Regarding configuration of network buffers: http://ci.apache.org/projects/flink/flink-docs-master/setup/config.html#configuring-the-network-buffers

With this high core count per machine, it is suggested to give the network layer quite some
memory for concurrent shuffles. If you can afford it, go as high as a couple of GBs. The formula
in the docs would suggest 262144 buffers, e.g. 8 GB.

If this is not an option, I would decrease the number of slots per task manager (e.g. to 32)
and reserve less buffers for the network stack.

Feel free to post any time when something comes up.

– Ufuk
Mime
View raw message