flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fabian Hueske <fhue...@gmail.com>
Subject Re: Flink's multi-user support
Date Wed, 29 Apr 2015 15:51:00 GMT
I agree that Flink's multi-user support is not very good at the moment.
However, dropping it completely instead of improving it would make Flink
setups on dedicated clusters quite useless, right?


2015-04-29 17:33 GMT+02:00 Maximilian Michels <mxm@apache.org>:

> Hi everyone,
>
> Currently Flink accepts jobs from multiple clients and executes them
> concurrently if the resource limits are not exceeded. However, the
> multi-user support is very poor. We don't support queuing of jobs and
> concurrent jobs have to share resources in a nice way. Otherwise, jobs will
> fail.
>
> Using YARN, we circumvent these problems because it provides a proper user
> and session management. I'm wondering now, should we get rid of the pseudo
> multi-user mode and just support one user per Flink cluster instance?
>
> Best,
> Max
>
> PS:
> This question came up when I was working on a pull request to support
> backtracking intermediate results. I need to hold a copy of the full
> previous execution graph to resume from old results. With multiple users,
> we have to build in some kind of session management to archive old
> execution graphs. Otherwise, they will consume too much memory in the job
> manager.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message