hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jian He <...@hortonworks.com>
Subject Re: Capacity Scheduler capacity vs. maximum-capacity
Date Thu, 20 Feb 2014 02:35:01 GMT
Yes, in the scenario you mentioned,  the scheduler will take away the 10%
from queue B and give it back to queue A


On Wed, Feb 19, 2014 at 5:35 PM, Alex Nastetsky <anastetsky@spryinc.com>wrote:

> Thanks Jian.
>
> In what sense is the "capacity" resources "guaranteed"?
>
> Let's say there are two queues at the root level, A and B. Both have
> capacity of 50% and max capacity of 70%. If queue A is only currently using
> 40% and queue B is using 50% and needs more, then it will be able to borrow
> 10% from queue A, so now queue A is using 40% and queue B is using 60%. But
> what happens if queue A now needs 50%? Will the scheduler take away the 10%
> from queue B and give it back to queue A even if queue B needs it? If not,
> it would seem that the scheduler is reneging on its guarantee.
>
>
> On Wed, Feb 19, 2014 at 6:01 PM, Jian He <jhe@hortonworks.com> wrote:
>
>> Hi Alex
>>
>> You can find good explanation from here:
>> http://hortonworks.com/blog/understanding-apache-hadoops-capacity-scheduler/
>> Short term: Capacity is the soft limit that queue is guaranteed for such
>> an amount of resource. For the purpose of necessary elasticity, queue can
>> go beyond capacity limit but can not go beyond Max-Capacity limit which is
>> the hard limit.
>>
>> Jian
>>
>>
>> On Wed, Feb 19, 2014 at 11:08 AM, Alex Nastetsky <anastetsky@spryinc.com>wrote:
>>
>>> Please help me understand how "capacity" and "maximum-capacity" on a
>>> queue work in the Capacity Scheduler.
>>>
>>> My understanding is that a queue is allocated "capacity" amount of
>>> resources, and if it needs more, it can "stretch" up to "maximum-capacity"
>>> resources.
>>>
>>> But if that's the case, why do we need "capacity" at all? It seems like
>>> "maximum-capacity" is the true limit and "capacity" is ignored?
>>>
>>> Thanks in advance,
>>> Alex.
>>>
>>
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>
>
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Mime
View raw message