flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stefano Baghino <stefano.bagh...@radicalbit.io>
Subject Re: Flink on YARN: long-running session vs. one-off jobs
Date Mon, 07 Mar 2016 14:14:48 GMT
Good, thank you for the explanation!

On Mon, Mar 7, 2016 at 2:38 PM, Maximilian Michels <mxm@apache.org> wrote:

> Hi Stefano,
>
> Essentially the Yarn Session is not much different from a per-job Yarn
> cluster. In either case, a Flink cluster is brought up with resources
> provided by Yarn. In case of the Yarn session this cluster doesn't do
> anything until a job is submitted. In case of the per-job Yarn
> cluster, a job is immediately submitted after startup and the cluster
> is shutdown once that job has been completed. That's all.
>
> We're currently working on integrating proper resource allocation into
> the JobManager. As of now, everything is static, i.e. the JobManager
> won't allocate more than the initial requested resources.
>
> Cheers,
> Max
>
> On Mon, Mar 7, 2016 at 1:38 PM, Stefano Baghino
> <stefano.baghino@radicalbit.io> wrote:
> > Hello everybody,
> >
> > I'm currently studying how the Flink/YARN integration works. Right now
> I'm a
> > little confused about the practical difference in having a long-running
> > session on which one deploys several jobs or deploying these jobs
> > individually.
> >
> > My intuition (which may not be correct) is that in the former ase a pool
> of
> > resources is allocated to the session and then the usage of these is
> handled
> > by the Flink JobManager, while in the latter case the resource
> allocation is
> > handled directly by YARN on a per-job basis. Am I right?
> >
> > If what I said is (more or less) right and apart from security concerns
> > (which have been discussed in a previous thread), are there any further
> > practical differences between having a long-running session or letting
> YARN
> > handle jobs?
> >
> > Thank you in advance!
> >
> > --
> > BR,
> > Stefano Baghino
> >
> > Software Engineer @ Radicalbit
>



-- 
BR,
Stefano Baghino

Software Engineer @ Radicalbit

Mime
View raw message