flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Maximilian Michels <...@apache.org>
Subject Re: Flink on YARN: long-running session vs. one-off jobs
Date Mon, 07 Mar 2016 13:38:45 GMT
Hi Stefano,

Essentially the Yarn Session is not much different from a per-job Yarn
cluster. In either case, a Flink cluster is brought up with resources
provided by Yarn. In case of the Yarn session this cluster doesn't do
anything until a job is submitted. In case of the per-job Yarn
cluster, a job is immediately submitted after startup and the cluster
is shutdown once that job has been completed. That's all.

We're currently working on integrating proper resource allocation into
the JobManager. As of now, everything is static, i.e. the JobManager
won't allocate more than the initial requested resources.


On Mon, Mar 7, 2016 at 1:38 PM, Stefano Baghino
<stefano.baghino@radicalbit.io> wrote:
> Hello everybody,
> I'm currently studying how the Flink/YARN integration works. Right now I'm a
> little confused about the practical difference in having a long-running
> session on which one deploys several jobs or deploying these jobs
> individually.
> My intuition (which may not be correct) is that in the former ase a pool of
> resources is allocated to the session and then the usage of these is handled
> by the Flink JobManager, while in the latter case the resource allocation is
> handled directly by YARN on a per-job basis. Am I right?
> If what I said is (more or less) right and apart from security concerns
> (which have been discussed in a previous thread), are there any further
> practical differences between having a long-running session or letting YARN
> handle jobs?
> Thank you in advance!
> --
> BR,
> Stefano Baghino
> Software Engineer @ Radicalbit

View raw message