flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From black chase <chaseandbl...@gmail.com>
Subject Re: How many task managers to launch for a job?
Date Fri, 24 May 2019 08:17:56 GMT
Hi Song,
Thank you for the clarification.
Now I know TaskManagers are automatically allocated. Yet, I am still not
very clear how the TMs are allocated.
I'm guessing the allocation process would be:
On the job side, I have a job with each operator parallelism=5. Since one
TaskManager has one slot. It means this job would need 5 TaskManagers
becase one pipeline would need one slot according to the task scheduling
policy.
On the Mesos side, let's say currently there are no available TaskManagers.
Then Mesos would spawn 5 new TaskManagers for this job. Is it?
If what I am guessing is right, then let's say we have 4 physical computing
nodes for the Flink TaskManagers. How would Mesos place the 5 new
TaskManagers into the 4 physical computing nodes? Is it juts a Round-Robin
fashion?
Kind regards
Chase



On Fri, May 24, 2019 at 4:10 AM Xintong Song <tonysong820@gmail.com> wrote:

> Hi black,
>
> If you are running Flink on Yarn or Mesos, Flink will automatically
> allocate resource and launch new TaskManagers as needed.
>
> If you are using Flink standalone mode, then the easiest way is to enable
> slot sharing and set all the vertices into the same group (which is by
> default). In that way, the total slots (or number of TaskManagers if you
> config on slot for each TaskManager) needed for running the job would be
> the maximum parallelism of the job graph vertices. Further information on
> slot sharing could be found here
> <https://ci.apache.org/projects/flink/flink-docs-release-1.8/concepts/runtime.html#task-slots-and-resources>
> .
>
> Thank you~
>
> Xintong Song
>
>
>
> On Thu, May 23, 2019 at 11:49 PM black chase <chaseandblack@gmail.com>
> wrote:
>
>>
>> Hi,
>>
>> I am redesigning the scheduler of the JobManager to place tasks of a job
>> across TaskManagers accroding to a scheduling policy.
>>
>> I am reading the Flip-6 proposal and found that the common case is "one
>> TaskManager launchs one slot", and "one Flink cluster serves one job". But
>> I did not find how many TaskManagers to launch in a computing node. Is
>> there any common practice for this ?
>>
>> --
>> Best Regards!
>> Pengcheng Duan
>>
>

-- 
Best Regards!

Mime
View raw message