flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vikram Saxena <vikram....@gmail.com>
Subject Re: Flink Queue Scheduling (JobManager)
Date Wed, 27 Apr 2016 14:16:58 GMT
Yes, I have tried using separate Yarn queues for this, but I have my
doubts.

Here is what I am trying to do:
I have 2 Flink Jobs

JobA : Regular Job running every x minutes.
JobB : User requested adhoc job

Tried1.

Sol A :
Have 2 queues on Yarn with 95 : 5 resource distribution

Sol B:
Have parallelism control the concurrent execution i.e. divide total task
slots in 95:5 ratio

In both the above solutions I would be dedicating a chunk of resources to
 Job B.
I want Job A to run will all my resources and only if Job B is submitted,
it should get some resources to process and not throw back
NoResourceException.

I hope I am able to explain my problem :)


VG,
VIkram

On Wed, Apr 27, 2016 at 2:13 PM, Flavio Pompermaier <pompermaier@okkam.it>
wrote:

> That would be definitely interesting but I think that at the moment the
> only way to achieve that is to exploit YARN for that..
> An integration with some job-workflow engine (like Apache Oozie and Apache
> Falcon) would also be very useful! I tried to wrote on Apache Oozie mailing
> list if there is any interest in integrating with apache Flink but I didn't
> receive any feedback yet :(
>
> Best,
> Flavio
>
> On Wed, Apr 27, 2016 at 1:00 PM, Vikram Saxena <vikram.mbt@gmail.com>
> wrote:
>
>> Hi
>>
>> I am reading and learning about Flink and I have tried to implement some
>> Flink Jobs.
>>
>> In my application I have 2 Flink Jobs which I want to run in parallel.
>> Of course, as I understand I can have the task slots divided so that each
>> one can run concurrently.
>>
>> But, is there a possibility for scheduling jobs only to be processed when
>> the JobManager has resources free(task slots) and not throw back
>> (NoResourceAvailableException) ??
>>
>> The reason I ask this is because I have 2 Flink Jobs, one which would be
>> always running in short intervals and other which would be in ad hoc basis.
>>
>> I do not want to have my task slots divided just for this ad hoc job,
>> So I wanted to submit this adhoc job and the Job Manager can schedule it
>> whenever it has free task slots ?
>>
>>
>> Cheers,
>> Vikram
>>
>
>

Mime
View raw message