hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ivan Tretyakov <itretya...@griddynamics.com>
Subject Re: Limit on total jobs running using fair scheduler
Date Thu, 21 Nov 2013 21:46:07 GMT
Thank you for your replies!

We are using MR version 1 and my question is regarding this version.

Omkar,
are you talking about MR1 or MR2? I didn't find property to limit number of
running jobs per queue for capacity scheduler using MR1. Did I wrong
somewhere? What exactly option do you mean?

Sandy,
thanks, I got it. But unfortunately we are using MR1 for now.


On Wed, Nov 20, 2013 at 2:12 AM, Sandy Ryza <sandy.ryza@cloudera.com> wrote:

> Unfortunately, this is not possible in the MR1 fair scheduler without
> setting the jobs for individual pools.  In MR2, fair scheduler hierarchical
> queues will allow setting maxRunningApps at the top of the hierarchy, which
> would have the effect you're looking for.
>
> -Sandy
>
>
> On Tue, Nov 19, 2013 at 2:01 PM, Omkar Joshi <ojoshi@hortonworks.com>wrote:
>
>> Not sure about the fair scheduler but in capacity scheduler you can
>> achieve this by controlling number of jobs/applications per queue.
>>
>> Thanks,
>> Omkar Joshi
>> *Hortonworks Inc.* <http://www.hortonworks.com>
>>
>>
>> On Tue, Nov 19, 2013 at 3:26 AM, Ivan Tretyakov <
>> itretyakov@griddynamics.com> wrote:
>>
>>> Hello!
>>>
>>> We are using CDH 4.1.1 (Version: 2.0.0-mr1-cdh4.1.1) and fair-scheduler.
>>> We need to limit total number of jobs which can run at the same time on
>>> cluster.
>>> I can see maxRunningJobs options but it sets limit for pool or user.
>>> We wouldn't like to limit each pool or user we just need to set limit on
>>> total number of jobs running.
>>>
>>> Is it possible to do it using fair scheduler?
>>> Can capacity scheduler help here?
>>> Maybe there are other options to achieve the goal?
>>>
>>> Thanks in advance!
>>>
>>> --
>>> Best Regards
>>> Ivan Tretyakov
>>>
>>>
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>
>
>


-- 
Best Regards
Ivan Tretyakov

Deployment Engineer
Grid Dynamics
+7 812 640 38 76
Skype: ivan.v.tretyakov
www.griddynamics.com
itretyakov@griddynamics.com

Mime
View raw message