flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yang Wang <danrtsey...@gmail.com>
Subject Re: How can TMs distribute evenly over Flink on YARN cluster?
Date Mon, 26 Aug 2019 10:21:25 GMT
Hi Qi Kang,

If you means to spread out all taskmanager evenly across the yarn cluster,
it is a pity that flink could do nothing.
Each per-job flink cluster is an individual application on the yarn
cluster, they do not know the existence of others.

Could share the yarn version? If it is above hadoop-3.x, then you should
set the
*yarn.scheduler.capacity.per-node-heartbeat.multiple-assignments-enabled=false*
to avoid assign multiple containers to one nodemanager in a hearbeat.


Best,
Yang

Qi Kang <miraisenshi@126.com> 于2019年8月26日周一 下午4:52写道:

> Hi,
>
>
> We got 3 Flink jobs running on a 10-node YARN cluster. The jobs were
> submitted in a per-job flavor, with same parallelism (10) and number of
> slots per TM (2).
>
> We originally assumed that TMs should automatically spread across the
> cluster, but what came out was just the opposite: All 5 TMs from one job
> simply went into one single node, thus leaving 7 nodes (almost) idle, and 3
> nodes under pressure.
>
> Is there some way to have those TMs evenly distributed? Many thanks.
>
>
>
>

Mime
View raw message