flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Metzger <rmetz...@apache.org>
Subject Re: How to choose the 'parallelism.default' value
Date Thu, 05 May 2016 11:20:17 GMT
Hi,

I think you've chosen a good initial value for the parallelism.
The higher the parallelism, the more network buffers are needed. I would
follow the recommendation from the exception and increase the number of
network buffers.

On Thu, May 5, 2016 at 11:23 AM, Punit Naik <naik.punit44@gmail.com> wrote:

> Hello
>
> I was running a program with 'parallelism.default' of 384 as I read in the
> documentation on Flink's official page that 'parallelism.default' is "the
> total number of CPUs in the cluster". I have four machines with 96 cores on
> each of them. So 96*4=384. But the program thew an error saying:
>
> Caused by: java.io.IOException: Insufficient number of network buffers:
> required 384, but only 298 available. The total number of network buffers
> is currently set to 2048. You can increase this number by setting the
> configuration key 'taskmanager.network.numberOfBuffers'.
>
> What does this mean? And how to choose a proper value for parallelism?
>
> --
> Thank You
>
> Regards
>
> Punit Naik
>

Mime
View raw message