flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Punit Naik <naik.puni...@gmail.com>
Subject How to choose the 'parallelism.default' value
Date Thu, 05 May 2016 09:23:52 GMT

I was running a program with 'parallelism.default' of 384 as I read in the
documentation on Flink's official page that 'parallelism.default' is "the
total number of CPUs in the cluster". I have four machines with 96 cores on
each of them. So 96*4=384. But the program thew an error saying:

Caused by: java.io.IOException: Insufficient number of network buffers:
required 384, but only 298 available. The total number of network buffers
is currently set to 2048. You can increase this number by setting the
configuration key 'taskmanager.network.numberOfBuffers'.

What does this mean? And how to choose a proper value for parallelism?

Thank You


Punit Naik

View raw message