flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stefano Baghino <stefano.bagh...@radicalbit.io>
Subject Re: Flink on YARN: long-running session vs. one-off jobs
Date Mon, 07 Mar 2016 17:37:09 GMT
One last question: running multiple jobs mean that each one has its own
JobManager, right?

On Mon, Mar 7, 2016 at 3:14 PM, Stefano Baghino <
stefano.baghino@radicalbit.io> wrote:

> Good, thank you for the explanation!
>
> On Mon, Mar 7, 2016 at 2:38 PM, Maximilian Michels <mxm@apache.org> wrote:
>
>> Hi Stefano,
>>
>> Essentially the Yarn Session is not much different from a per-job Yarn
>> cluster. In either case, a Flink cluster is brought up with resources
>> provided by Yarn. In case of the Yarn session this cluster doesn't do
>> anything until a job is submitted. In case of the per-job Yarn
>> cluster, a job is immediately submitted after startup and the cluster
>> is shutdown once that job has been completed. That's all.
>>
>> We're currently working on integrating proper resource allocation into
>> the JobManager. As of now, everything is static, i.e. the JobManager
>> won't allocate more than the initial requested resources.
>>
>> Cheers,
>> Max
>>
>> On Mon, Mar 7, 2016 at 1:38 PM, Stefano Baghino
>> <stefano.baghino@radicalbit.io> wrote:
>> > Hello everybody,
>> >
>> > I'm currently studying how the Flink/YARN integration works. Right now
>> I'm a
>> > little confused about the practical difference in having a long-running
>> > session on which one deploys several jobs or deploying these jobs
>> > individually.
>> >
>> > My intuition (which may not be correct) is that in the former ase a
>> pool of
>> > resources is allocated to the session and then the usage of these is
>> handled
>> > by the Flink JobManager, while in the latter case the resource
>> allocation is
>> > handled directly by YARN on a per-job basis. Am I right?
>> >
>> > If what I said is (more or less) right and apart from security concerns
>> > (which have been discussed in a previous thread), are there any further
>> > practical differences between having a long-running session or letting
>> YARN
>> > handle jobs?
>> >
>> > Thank you in advance!
>> >
>> > --
>> > BR,
>> > Stefano Baghino
>> >
>> > Software Engineer @ Radicalbit
>>
>
>
>
> --
> BR,
> Stefano Baghino
>
> Software Engineer @ Radicalbit
>



-- 
BR,
Stefano Baghino

Software Engineer @ Radicalbit

Mime
View raw message