flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Flavio Pompermaier <pomperma...@okkam.it>
Subject Re: Flink's multi-user support
Date Thu, 30 Apr 2015 09:09:17 GMT
There was an attempt to build such a queue during the Dopa project when
Flink was still Stratosphere.
Probably it could be a good idea to collect the good and bad things learned
from it to start designing the new scheduler :)

On Thu, Apr 30, 2015 at 10:08 AM, Stephan Ewen <sewen@apache.org> wrote:

> Most components are written multi-job aware.
>
> The only thing that is not in there right now is scheduling policies for
> fair resource sharing. This is important in shared clusters.
>
> Since YARN implements all those things (various job queues with different
> priorities/policies etc), I suggest to not try and re-build it in Flink and
> simply declare a JobManager a "single-job-at-a-time" manager. You can still
> run an interactive session with many jobs one after another.
>
>
> On Wed, Apr 29, 2015 at 7:07 PM, Maximilian Michels <mxm@apache.org>
> wrote:
>
> > >
> > > However, dropping it completely instead of improving it would make
> Flink
> > > setups on dedicated clusters quite useless, right?
> > >
> >
> > Not really, because you could also use YARN on dedicated clusters for
> > proper multi-user support.
> >
> > On Wed, Apr 29, 2015 at 5:51 PM, Fabian Hueske <fhueske@gmail.com>
> wrote:
> >
> > > I agree that Flink's multi-user support is not very good at the moment.
> > > However, dropping it completely instead of improving it would make
> Flink
> > > setups on dedicated clusters quite useless, right?
> > >
> > >
> > > 2015-04-29 17:33 GMT+02:00 Maximilian Michels <mxm@apache.org>:
> > >
> > > > Hi everyone,
> > > >
> > > > Currently Flink accepts jobs from multiple clients and executes them
> > > > concurrently if the resource limits are not exceeded. However, the
> > > > multi-user support is very poor. We don't support queuing of jobs and
> > > > concurrent jobs have to share resources in a nice way. Otherwise,
> jobs
> > > will
> > > > fail.
> > > >
> > > > Using YARN, we circumvent these problems because it provides a proper
> > > user
> > > > and session management. I'm wondering now, should we get rid of the
> > > pseudo
> > > > multi-user mode and just support one user per Flink cluster instance?
> > > >
> > > > Best,
> > > > Max
> > > >
> > > > PS:
> > > > This question came up when I was working on a pull request to support
> > > > backtracking intermediate results. I need to hold a copy of the full
> > > > previous execution graph to resume from old results. With multiple
> > users,
> > > > we have to build in some kind of session management to archive old
> > > > execution graphs. Otherwise, they will consume too much memory in the
> > job
> > > > manager.
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message