flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sourigna Phetsarath <gna.phetsar...@teamaol.com>
Subject Re: Setting taskmanager.network.numberOfBuffers does not seem to have an affect - Flink 0.10.2
Date Fri, 18 Mar 2016 20:38:28 GMT
It's batch.

Reading a few thousand Avro files from S3 with a parallelism of 320.  It's
just failing on the read right now and not making to the next operations.

Here's a screenshot of the flink dashboard.  It usually fails in the 1st
task.

Cheers,
Gna

On Fri, Mar 18, 2016 at 4:03 PM, Ufuk Celebi <uce@apache.org> wrote:

> Hey Souringa,
>
> could you provide some context about the program you are running?
>
> Is it batch or streaming? What is the parallelism? How many operators are
> you running?
>
> Thanks for reporting the issue. I think we will figure it out once you
> provide further information. :-)
>
> – Ufuk
>
> On Fri, Mar 18, 2016 at 7:15 PM, Sourigna Phetsarath <
> gna.phetsarath@teamaol.com> wrote:
>
>> All:
>>
>> Flink Version 0.10.2
>>
>> The number that I set for *taskmanager.network.numberOfBuffers* doesn't
>> seem to have any affect, even if I set it to a very high number. There
>> might be a race condition here where the upper bound is not enforced or
>> computer correctly.
>>
>> java.io.IOException: Insufficient number of network buffers: required
>> *320*, but only *32* available. The total number of network buffers is
>> currently set to *36864*. You can increase this number by setting the
>> configuration key 'taskmanager.network.numberOfBuffers'.
>> at
>> org.apache.flink.runtime.io.network.buffer.NetworkBufferPool.createBufferPool(NetworkBufferPool.java:196)
>> at
>> org.apache.flink.runtime.io.network.NetworkEnvironment.registerTask(NetworkEnvironment.java:296)
>> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:488)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> Has anyone else encountered this issue?
>>
>> Thanks in advance for any help that anyone can provide.
>> --
>>
>>
>> *Gna Phetsarath*System Architect // AOL Platforms // Data Services //
>> Applied Research Chapter
>> 770 Broadway, 5th Floor, New York, NY 10003
>> o: 212.402.4871 // m: 917.373.7363
>> vvmr: 8890237 aim: sphetsarath20 t: @sourigna
>>
>> * <http://www.aolplatforms.com>*
>>
>
>


-- 


*Gna Phetsarath*System Architect // AOL Platforms // Data Services //
Applied Research Chapter
770 Broadway, 5th Floor, New York, NY 10003
o: 212.402.4871 // m: 917.373.7363
vvmr: 8890237 aim: sphetsarath20 t: @sourigna

* <http://www.aolplatforms.com>*

Mime
View raw message