flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Metzger <rmetz...@apache.org>
Subject Re: Adjusting number of YARN containers
Date Wed, 09 Sep 2015 12:29:59 GMT
Hi,

Currently, Flink does not support automatic scaling of the YARN containers.
There are certainly plans to add this feature:
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/add-some-new-api-to-the-scheduler-in-the-job-manager-tp7153p7448.html

Adding an API for manually starting and stopping containers is something we
can do very easily, something dynamic is a bit harder.
The feature is something I was always interested to work on, but nobody
really asked for it. But it seems that more and more people are asking for
it.
If this is a requirement for you to use Flink, I'm certainly willing to add
the feature into Flink.

Best,
Robert




On Wed, Sep 9, 2015 at 11:45 AM, Peter Voß <pv@datameer.com> wrote:

> Hi,
>
> I have started a Flink YARN session using yarn-session.sh and the
> configuration of number of YARN container seems to be pretty static.
>
> Is it possible to have Flink adjust the number of containers depending on
> the actual workload. E.g. stop containers that are idle for too long and
> start them up again when more jobs get submitted?
> Or are there any plans to have more dynamic resource management?
>
> Thanks,
> —Peter

Mime
View raw message