hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arun C Murthy <...@hortonworks.com>
Subject Re: Capacity Scheduler on YARN
Date Tue, 30 Jul 2013 11:52:10 GMT

 The CS, by default, will cap a single user at the queue's capacity (not max-capacity) - this
is why you are seeing that any given user cannot go beyond the queue's capacity… in other
words, you will need more than one user to utilize max-capacity of the queue.

 If you want to allow this behavior, you can set user-limit-factor (a multiple of queue-capacity),
see yarn.scheduler.capacity.<queue-path>.user-limit-factor in:


On Jul 30, 2013, at 2:33 AM, Hans-Peter Zorn <hpzorn@gmail.com> wrote:

> Hi,
> I would like to use the capacity scheduler to implement the following requirements.
> 3 groups of users:
> * 2 of them are entitled to 50% of cluster capacity each but may use excess capacity
if available
> * the third group only may use capacity that is currently not used by any of the other
> I set up three queues:
> yarn.scheduler.capacity.root.a.capacity 50
> yarn.scheduler.capacity.root.a.maximum-capacity 100
> yarn.scheduler.capacity.root.b.capacity 49
> yarn.scheduler.capacity.root.b.maximum-capacity 100
> yarn.scheduler.capacity.root.c.capacity 1
> yarn.scheduler.capacity.root.c.maximum-capacity 100
> These values are active, as they show up in the scheduler web gui.
> However, users of each queue are only able to use the assigned capacity, the value
> of maximum-capacity has no effect.
> Did I misunderstand the concept behind the maximum-capacity setting?
> I am using hadoop as shipped with Cloudera CDH4.3  (2.0.0-cdh4.3.0). I asked this question
already on cdh-users, but the capacity scheduler seems not to be used so much with cdh I guess.
> Or is it possible to implement this using a different scheduler?
> Best & thanks,
> Hans-Peter

Arun C. Murthy
Hortonworks Inc.

View raw message