flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Metzger <rmetz...@apache.org>
Subject Re: Adjusting number of YARN containers
Date Wed, 09 Sep 2015 12:29:59 GMT

Currently, Flink does not support automatic scaling of the YARN containers.
There are certainly plans to add this feature:

Adding an API for manually starting and stopping containers is something we
can do very easily, something dynamic is a bit harder.
The feature is something I was always interested to work on, but nobody
really asked for it. But it seems that more and more people are asking for
If this is a requirement for you to use Flink, I'm certainly willing to add
the feature into Flink.


On Wed, Sep 9, 2015 at 11:45 AM, Peter Voß <pv@datameer.com> wrote:

> Hi,
> I have started a Flink YARN session using yarn-session.sh and the
> configuration of number of YARN container seems to be pretty static.
> Is it possible to have Flink adjust the number of containers depending on
> the actual workload. E.g. stop containers that are idle for too long and
> start them up again when more jobs get submitted?
> Or are there any plans to have more dynamic resource management?
> Thanks,
> —Peter

View raw message