hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alex Nastetsky <anastet...@spryinc.com>
Subject Re: Capacity Scheduler capacity vs. maximum-capacity
Date Thu, 20 Feb 2014 03:03:45 GMT
Thanks, makes sense now.
On Feb 19, 2014 9:35 PM, "Jian He" <jhe@hortonworks.com> wrote:

> Yes, in the scenario you mentioned,  the scheduler will take away the 10%
> from queue B and give it back to queue A
>
>
> On Wed, Feb 19, 2014 at 5:35 PM, Alex Nastetsky <anastetsky@spryinc.com>wrote:
>
>> Thanks Jian.
>>
>> In what sense is the "capacity" resources "guaranteed"?
>>
>> Let's say there are two queues at the root level, A and B. Both have
>> capacity of 50% and max capacity of 70%. If queue A is only currently using
>> 40% and queue B is using 50% and needs more, then it will be able to borrow
>> 10% from queue A, so now queue A is using 40% and queue B is using 60%. But
>> what happens if queue A now needs 50%? Will the scheduler take away the 10%
>> from queue B and give it back to queue A even if queue B needs it? If not,
>> it would seem that the scheduler is reneging on its guarantee.
>>
>>
>> On Wed, Feb 19, 2014 at 6:01 PM, Jian He <jhe@hortonworks.com> wrote:
>>
>>> Hi Alex
>>>
>>> You can find good explanation from here:
>>> http://hortonworks.com/blog/understanding-apache-hadoops-capacity-scheduler/
>>> Short term: Capacity is the soft limit that queue is guaranteed for such
>>> an amount of resource. For the purpose of necessary elasticity, queue can
>>> go beyond capacity limit but can not go beyond Max-Capacity limit which is
>>> the hard limit.
>>>
>>> Jian
>>>
>>>
>>> On Wed, Feb 19, 2014 at 11:08 AM, Alex Nastetsky <anastetsky@spryinc.com
>>> > wrote:
>>>
>>>> Please help me understand how "capacity" and "maximum-capacity" on a
>>>> queue work in the Capacity Scheduler.
>>>>
>>>> My understanding is that a queue is allocated "capacity" amount of
>>>> resources, and if it needs more, it can "stretch" up to "maximum-capacity"
>>>> resources.
>>>>
>>>> But if that's the case, why do we need "capacity" at all? It seems like
>>>> "maximum-capacity" is the true limit and "capacity" is ignored?
>>>>
>>>> Thanks in advance,
>>>> Alex.
>>>>
>>>
>>>
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity
>>> to which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender immediately
>>> and delete it from your system. Thank You.
>>
>>
>>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.

Mime
View raw message