hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Varun Saxena <vsaxena.va...@gmail.com>
Subject Re: yarn.nodemanager.resource.cpu-vcores vs yarn.scheduler.maximum-allocation-vcores
Date Sun, 23 Aug 2015 19:40:08 GMT
The fix would be released in next version(2.8.0).
I had checked the code to find out the default value and then found it
fixed in documentation(configuration list).

As this is an unreleased version, a URL link (of the form
https://hadoop.apache.org/docs/r2.7.1/hadoop-yarn/hadoop-yarn-common/yarn-default.xml)
may not be available AFAIK,
However, this XML(yarn-default.xml) can be checked online in git repository.

Associated JIRA which fixes this is
https://issues.apache.org/jira/browse/YARN-3823

Regards,
Varun Saxena.

On Mon, Aug 24, 2015 at 12:53 AM, Pedro Magalhaes <pedrorjbr@gmail.com>
wrote:

> Thanks Varun!
> Could plz send me the link with the fixed?
>
> On Sun, Aug 23, 2015 at 2:20 PM, Varun Saxena <vsaxena.varun@gmail.com>
> wrote:
>
>> Hi Pedro,
>>
>> Real default value of yarn.scheduler.maximum-allocation-vcores is 4.
>> The value of 32 is actually a documentation issue and has been fixed
>> recently.
>>
>> Regards,
>> Varun Saxena.
>>
>>
>> On Sun, Aug 23, 2015 at 10:39 PM, Pedro Magalhaes <pedrorjbr@gmail.com>
>> wrote:
>>
>>> Varun,
>>> Thanks for the reply. I undestand the arn.scheduler.maximum-
>>> allocation-vcores parameter. I just asking why the default parameter is
>>> yarn.scheduler.maximum-allocation-vcores=32. And
>>> yarn.nodemanager.resource.cpu-vcores=8.
>>>
>>> In my opinion, if the yarn.scheduler.maximun-allocation-vcore is 32 tby
>>> default the yarn.nodemanager.resource.cpu-vcores  would be equal or greater
>>> than 32, by default.
>>> Is this make sense?
>>>
>>>
>>>
>>>
>>> On Sun, Aug 23, 2015 at 2:00 PM, Varun Saxena <vsaxena.varun@gmail.com>
>>> wrote:
>>>
>>>> Hi Pedro,
>>>>
>>>> Actual allocation would depend on the total resource capability
>>>> advertised by NM while registering with RM.
>>>>
>>>> yarn.scheduler.maximum-allocation-vcores merely puts an upper cap on number
of vcores which can be allocated by RM i.e. any Resource request/ask from AM which asks for
vcores > 32(default value) for a container, will be normalized back to 32.
>>>>
>>>> If there is no such node available, this allocation will not be fulfilled.
>>>>
>>>> yarn.scheduler.maximum-allocation-vcores will be configured in resource
>>>> manager and hence will be common for a cluster which can possibly have
>>>> multiple nodes with heterogeneous resource capabilities
>>>>
>>>> yarn.nodemanager.resource.cpu-vcores on the other hand will have to be
>>>> configured as per resource capability of that particular node.
>>>>
>>>> Recently there has been work done to automatically get memory and CPU
>>>> information from underlying OS(supported OS being Linux and Windows) if
>>>> configured to do so. This change would be available in 2.8
>>>> I hope this answers your question.
>>>>
>>>> Regards,
>>>> Varun Saxena.
>>>>
>>>> On Sun, Aug 23, 2015 at 9:40 PM, Pedro Magalhaes <pedrorjbr@gmail.com>
>>>> wrote:
>>>>
>>>>> I was looking at default parameters for:
>>>>>
>>>>> yarn.nodemanager.resource.cpu-vcores = 8
>>>>> yarn.scheduler.maximum-allocation-vcores = 32
>>>>>
>>>>> For me this two parameters as default doesnt make any sense.
>>>>>
>>>>> The first one say "the number of CPU cores that can be allocated for
>>>>> containers." (I imagine that is vcore) The seconds says: "The maximum
>>>>> allocation for every container request at the RM". In my opinion, the
>>>>> second one must be equal or less than the first one.
>>>>>
>>>>> How can allocate 32 vcores for a container if i have only 8 cores
>>>>> available per container?
>>>>>
>>>>
>>>>
>>>
>>
>

Mime
View raw message