flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Maximilian Michels <...@apache.org>
Subject Re: add some new api to the scheduler in the job manager
Date Mon, 27 Jul 2015 10:07:22 GMT
Hi MaGuoWei,

If I understand correctly, you're are looking for a way to have a job
manager (master) contentiously running which requests YARN containers for
the task managers (workers) on the fly. This is currently not supported by
Flink although you can add or remove task managers while the cluster is
running. The job manager distributes task to the task manager which are
available at scheduling time.

As Aljoscha mentioned, the most flexible way of sharing resources in a YARN
environment is to start a per-job cluster for each job.

Cheers,
Max

On Mon, Jul 27, 2015 at 11:19 AM, MaGuoWei <maguowei@outlook.com> wrote:

> It is great! Is there any document? I am very interested in this.
> thanks
>
>
> > From: aljoscha@apache.org
> > Date: Mon, 27 Jul 2015 05:14:00 +0000
> > Subject: Re: add some new api to the scheduler in the job manager
> > To: dev@flink.apache.org
> >
> > Hi,
> > I think for more details on giving back resources of a running cluster we
> > have to wait for Robert's opinion. In the mean time, you can also just
> run
> > a single job that will bring up some yarn containers and then release
> them
> > afterward using this:
> >
> https://ci.apache.org/projects/flink/flink-docs-release-0.9/setup/yarn_setup.html#run-a-single-flink-job-on-hadoop-yarn
> >
> > Cheers,
> > Aljoscha
> >
> > On Mon, 27 Jul 2015 at 03:51 MaGuoWei <maguowei@outlook.com> wrote:
> >
> > > hi guysFlink must take over all the resources all the time. That wastes
> > > resources sometime especially  in the sharing cluster.For example when
> > > using Flink on Yarn the resource can't be returned back to RM even if
> no
> > > graph is running.So I want to know: Is it possible to add some apis in
> the
> > > scheduler to request the resource(slot) and release the resource(slot)?
> > > These apis can be customized in the different circumstance.
> > > thanks a lot!
> > >
> > >
> > >
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message