flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From suraj7 <su...@amagi.com>
Subject Re: Flink TaskManagers do not start until job is submitted in YARN
Date Wed, 26 Sep 2018 08:02:25 GMT
Hi Till,

What I was ideally looking for was to have a completely managed service for
Flink via AWS EMR in which YARN Cluster would be completely dedicated to
only one Flink Session and as the EMR scales in and out, EMR/YARN would
add/remove TMs accordingly. I could then get the value of total task slots
across all running TMs from Flink REST API and change my Job parallelism
accordingly. 

As I understand, this kind of feature is not currently available and new TMs
will not be started as EMR scales out. The only way to get new TMs would be
to scale the EMR cluster to the required size and change Job parallelism and
expect Flink and YARN to take care of spawning new TMs with Flink's Resource
elasticity. 

So, now we planning to poll YARN REST API to get current Active nodes which
will be equal to the number of TMs(Flink will be configured to run 1 TM per
EC2 instance) the cluster is capable of running and then modify Job
parallelism accordingly. Can you validate this strategy and/or suggest
something better?

Thanks,
Suraj



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Mime
View raw message