flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From black chase <chaseandbl...@gmail.com>
Subject Re: How many task managers to launch for a job?
Date Fri, 24 May 2019 09:28:33 GMT
Yes true. I am trying to figure out how the TaskManagers are distributed
across physical machines by Mesos and YARN. Maybe I shoud start a new
thread for help.
Thank you Song
Best,
Pengcheng

On Fri, May 24, 2019 at 10:52 AM Xintong Song <tonysong820@gmail.com> wrote:

> As far as I know, Flink does not have any requirements on how the
> TaskManagers are distributed across physical machines. So I think it really
> depends on the scheduling policy of the Mesos cluster. I'm not an expert on
> Mesos, so correct me if I was wrong.
>
> Thank you~
>
> Xintong Song
>
>
>
> On Fri, May 24, 2019 at 4:18 PM black chase <chaseandblack@gmail.com>
> wrote:
>
>> Hi Song,
>> Thank you for the clarification.
>> Now I know TaskManagers are automatically allocated. Yet, I am still not
>> very clear how the TMs are allocated.
>> I'm guessing the allocation process would be:
>> On the job side, I have a job with each operator parallelism=5. Since one
>> TaskManager has one slot. It means this job would need 5 TaskManagers
>> becase one pipeline would need one slot according to the task scheduling
>> policy.
>> On the Mesos side, let's say currently there are no available
>> TaskManagers. Then Mesos would spawn 5 new TaskManagers for this job. Is it?
>> If what I am guessing is right, then let's say we have 4 physical
>> computing nodes for the Flink TaskManagers. How would Mesos place the 5 new
>> TaskManagers into the 4 physical computing nodes? Is it juts a Round-Robin
>> fashion?
>> Kind regards
>> Chase
>>
>>
>>
>> On Fri, May 24, 2019 at 4:10 AM Xintong Song <tonysong820@gmail.com>
>> wrote:
>>
>>> Hi black,
>>>
>>> If you are running Flink on Yarn or Mesos, Flink will automatically
>>> allocate resource and launch new TaskManagers as needed.
>>>
>>> If you are using Flink standalone mode, then the easiest way is to
>>> enable slot sharing and set all the vertices into the same group (which is
>>> by default). In that way, the total slots (or number of TaskManagers if you
>>> config on slot for each TaskManager) needed for running the job would be
>>> the maximum parallelism of the job graph vertices. Further information on
>>> slot sharing could be found here
>>> <https://ci.apache.org/projects/flink/flink-docs-release-1.8/concepts/runtime.html#task-slots-and-resources>
>>> .
>>>
>>> Thank you~
>>>
>>> Xintong Song
>>>
>>>
>>>
>>> On Thu, May 23, 2019 at 11:49 PM black chase <chaseandblack@gmail.com>
>>> wrote:
>>>
>>>>
>>>> Hi,
>>>>
>>>> I am redesigning the scheduler of the JobManager to place tasks of a
>>>> job across TaskManagers accroding to a scheduling policy.
>>>>
>>>> I am reading the Flip-6 proposal and found that the common case is "one
>>>> TaskManager launchs one slot", and "one Flink cluster serves one job". But
>>>> I did not find how many TaskManagers to launch in a computing node. Is
>>>> there any common practice for this ?
>>>>
>>>> --
>>>> Best Regards!
>>>> Pengcheng Duan
>>>>
>>>
>>
>> --
>> Best Regards!
>>
>

-- 
Best Regards!

Mime
View raw message