flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vijay Balakrishnan <bvija...@gmail.com>
Subject Re: Flink Dashboard UI Tasks hard limit
Date Fri, 22 May 2020 17:10:54 GMT
Hi Xintong,
Thx for your reply.  Increasing network memory buffers (fraction, min, max)
seems to increase tasks slightly.

Streaming job
Standalone

Vijay

On Fri, May 22, 2020 at 2:49 AM Xintong Song <tonysong820@gmail.com> wrote:

> Hi Vijay,
>
> I don't think your problem is related to number of opening files. The
> parallelism of your job is decided before actually tries to open the files.
> And if the OS limit for opening files is reached, you should see a job
> execution failure, instead of a success execution with a lower parallelism.
>
> Could you share some more information about your use case?
>
>    - What kind of job are your executing? Is it a streaming or batch
>    processing job?
>    - Which Flink deployment do you use? Standalone? Yarn?
>    - It would be helpful if you can share the Flink logs.
>
>
> Thank you~
>
> Xintong Song
>
>
>
> On Wed, May 20, 2020 at 11:50 PM Vijay Balakrishnan <bvijaykr@gmail.com>
> wrote:
>
>> Hi,
>> I have increased the number of slots available but the Job is not using
>> all the slots but runs into this approximate 18000 Tasks limit. Looking
>> into the source code, it seems to be opening file -
>> https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/api/common/io/FileOutputFormat.java#L203
>> So, do I have to tune the ulimit or something similar at the Ubuntu O/S
>> level to increase number of tasks available ? What I am confused about is
>> the ulimit is per machine but the ExecutionGraph is across many machines ?
>> Please pardon my ignorance here. Does number of tasks equate to number of
>> open files. I am using 15 slots per TaskManager on AWS m5.4xlarge which has
>> 16 vCPUs.
>>
>> TIA.
>>
>> On Tue, May 19, 2020 at 3:22 PM Vijay Balakrishnan <bvijaykr@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> Flink Dashboard UI seems to show tasks having a hard limit for Tasks
>>> column around 18000 on a Ubuntu Linux box.
>>> I kept increasing the number of slots per task manager to 15 and number
>>> of slots increased to 705 but the slots to tasks
>>> stayed at around 18000. Below 18000 tasks, the Flink Job is able to
>>> start up.
>>> Even though I increased the number of slots, it still works when 312
>>> slots are being used.
>>>
>>> taskmanager.numberOfTaskSlots: 15
>>>
>>> What knob can I tune to increase the number of Tasks ?
>>>
>>> Pls find attached the Flink Dashboard UI.
>>>
>>> TIA,
>>>
>>>

Mime
View raw message