flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Metzger <rmetz...@apache.org>
Subject Re: add some new api to the scheduler in the job manager
Date Mon, 27 Jul 2015 12:01:11 GMT
Hi MaGuoWei,

would you like to have done automatically by Flink or based on some user
input?
Adding commands to the ./bin/yarn-session.sh to change the cluster size is
quite easy. However, reducing the cluster size while a job is running will
fail the job.

Making this automatically is much harder.

On Mon, Jul 27, 2015 at 12:07 PM, Maximilian Michels <mxm@apache.org> wrote:

> Hi MaGuoWei,
>
> If I understand correctly, you're are looking for a way to have a job
> manager (master) contentiously running which requests YARN containers for
> the task managers (workers) on the fly. This is currently not supported by
> Flink although you can add or remove task managers while the cluster is
> running. The job manager distributes task to the task manager which are
> available at scheduling time.
>
> As Aljoscha mentioned, the most flexible way of sharing resources in a YARN
> environment is to start a per-job cluster for each job.
>
> Cheers,
> Max
>
> On Mon, Jul 27, 2015 at 11:19 AM, MaGuoWei <maguowei@outlook.com> wrote:
>
> > It is great! Is there any document? I am very interested in this.
> > thanks
> >
> >
> > > From: aljoscha@apache.org
> > > Date: Mon, 27 Jul 2015 05:14:00 +0000
> > > Subject: Re: add some new api to the scheduler in the job manager
> > > To: dev@flink.apache.org
> > >
> > > Hi,
> > > I think for more details on giving back resources of a running cluster
> we
> > > have to wait for Robert's opinion. In the mean time, you can also just
> > run
> > > a single job that will bring up some yarn containers and then release
> > them
> > > afterward using this:
> > >
> >
> https://ci.apache.org/projects/flink/flink-docs-release-0.9/setup/yarn_setup.html#run-a-single-flink-job-on-hadoop-yarn
> > >
> > > Cheers,
> > > Aljoscha
> > >
> > > On Mon, 27 Jul 2015 at 03:51 MaGuoWei <maguowei@outlook.com> wrote:
> > >
> > > > hi guysFlink must take over all the resources all the time. That
> wastes
> > > > resources sometime especially  in the sharing cluster.For example
> when
> > > > using Flink on Yarn the resource can't be returned back to RM even if
> > no
> > > > graph is running.So I want to know: Is it possible to add some apis
> in
> > the
> > > > scheduler to request the resource(slot) and release the
> resource(slot)?
> > > > These apis can be customized in the different circumstance.
> > > > thanks a lot!
> > > >
> > > >
> > > >
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message