hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alex Nastetsky <anastet...@spryinc.com>
Subject Re: Capacity Scheduler capacity vs. maximum-capacity
Date Thu, 20 Feb 2014 01:35:19 GMT
Thanks Jian.

In what sense is the "capacity" resources "guaranteed"?

Let's say there are two queues at the root level, A and B. Both have
capacity of 50% and max capacity of 70%. If queue A is only currently using
40% and queue B is using 50% and needs more, then it will be able to borrow
10% from queue A, so now queue A is using 40% and queue B is using 60%. But
what happens if queue A now needs 50%? Will the scheduler take away the 10%
from queue B and give it back to queue A even if queue B needs it? If not,
it would seem that the scheduler is reneging on its guarantee.

On Wed, Feb 19, 2014 at 6:01 PM, Jian He <jhe@hortonworks.com> wrote:

> Hi Alex
> You can find good explanation from here:
> http://hortonworks.com/blog/understanding-apache-hadoops-capacity-scheduler/
> Short term: Capacity is the soft limit that queue is guaranteed for such
> an amount of resource. For the purpose of necessary elasticity, queue can
> go beyond capacity limit but can not go beyond Max-Capacity limit which is
> the hard limit.
> Jian
> On Wed, Feb 19, 2014 at 11:08 AM, Alex Nastetsky <anastetsky@spryinc.com>wrote:
>> Please help me understand how "capacity" and "maximum-capacity" on a
>> queue work in the Capacity Scheduler.
>> My understanding is that a queue is allocated "capacity" amount of
>> resources, and if it needs more, it can "stretch" up to "maximum-capacity"
>> resources.
>> But if that's the case, why do we need "capacity" at all? It seems like
>> "maximum-capacity" is the true limit and "capacity" is ignored?
>> Thanks in advance,
>> Alex.
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.

View raw message