hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pedro Magalhaes <pedror...@gmail.com>
Subject Re: yarn.nodemanager.resource.cpu-vcores vs yarn.scheduler.maximum-allocation-vcores
Date Sun, 23 Aug 2015 20:18:19 GMT
So if I have nodes with different yarn.nodemanager.resource.cpu-vcores and
yarn.nodemanager.resource.memory-mb, the configuration will show me all
different values? Or will just show me just the default values?




On Sun, Aug 23, 2015 at 4:56 PM, Varun Saxena <vsaxena.varun@gmail.com>
wrote:

> This configuration is read and used by NodeManager, on whichever node its
> running.
> If it is not configured, default value will be taken.
>
> Regards,
> Varun Saxena.
>
> On Mon, Aug 24, 2015 at 1:21 AM, Pedro Magalhaes <pedrorjbr@gmail.com>
> wrote:
>
>> Thanks Varun! Like we say in Brazil.  "U are the guy!" (Você é o cara!)
>>
>> I have another question. You said that:
>> "yarn.nodemanager.resource.cpu-vcores on the other hand will have to be
>> configured as per resource capability of that particular node. "
>>
>> I get the configuration from my job and printed it:
>> yarn.nodemanager.resource.cpu-vcores 8
>> yarn.nodemanager.resource.memory-mb 8192
>>
>> So how does hadoop get this property if it is per node? Does it get the
>> minimum of all nodes? Thanks again!
>>
>>
>>
>> On Sun, Aug 23, 2015 at 4:40 PM, Varun Saxena <vsaxena.varun@gmail.com>
>> wrote:
>>
>>> The fix would be released in next version(2.8.0).
>>> I had checked the code to find out the default value and then found it
>>> fixed in documentation(configuration list).
>>>
>>> As this is an unreleased version, a URL link (of the form
>>> https://hadoop.apache.org/docs/r2.7.1/hadoop-yarn/hadoop-yarn-common/yarn-default.xml)
>>> may not be available AFAIK,
>>> However, this XML(yarn-default.xml) can be checked online in git
>>> repository.
>>>
>>> Associated JIRA which fixes this is
>>> https://issues.apache.org/jira/browse/YARN-3823
>>>
>>> Regards,
>>> Varun Saxena.
>>>
>>> On Mon, Aug 24, 2015 at 12:53 AM, Pedro Magalhaes <pedrorjbr@gmail.com>
>>> wrote:
>>>
>>>> Thanks Varun!
>>>> Could plz send me the link with the fixed?
>>>>
>>>> On Sun, Aug 23, 2015 at 2:20 PM, Varun Saxena <vsaxena.varun@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Pedro,
>>>>>
>>>>> Real default value of yarn.scheduler.maximum-allocation-vcores is 4.
>>>>> The value of 32 is actually a documentation issue and has been fixed
>>>>> recently.
>>>>>
>>>>> Regards,
>>>>> Varun Saxena.
>>>>>
>>>>>
>>>>> On Sun, Aug 23, 2015 at 10:39 PM, Pedro Magalhaes <pedrorjbr@gmail.com
>>>>> > wrote:
>>>>>
>>>>>> Varun,
>>>>>> Thanks for the reply. I undestand the arn.scheduler.maximum-
>>>>>> allocation-vcores parameter. I just asking why the default parameter
>>>>>> is yarn.scheduler.maximum-allocation-vcores=32. And
>>>>>> yarn.nodemanager.resource.cpu-vcores=8.
>>>>>>
>>>>>> In my opinion, if the yarn.scheduler.maximun-allocation-vcore is
32
>>>>>> tby default the yarn.nodemanager.resource.cpu-vcores  would be equal
or
>>>>>> greater than 32, by default.
>>>>>> Is this make sense?
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Sun, Aug 23, 2015 at 2:00 PM, Varun Saxena <
>>>>>> vsaxena.varun@gmail.com> wrote:
>>>>>>
>>>>>>> Hi Pedro,
>>>>>>>
>>>>>>> Actual allocation would depend on the total resource capability
>>>>>>> advertised by NM while registering with RM.
>>>>>>>
>>>>>>> yarn.scheduler.maximum-allocation-vcores merely puts an upper
cap on number of vcores which can be allocated by RM i.e. any Resource request/ask from AM
which asks for vcores > 32(default value) for a container, will be normalized back to 32.
>>>>>>>
>>>>>>> If there is no such node available, this allocation will not
be fulfilled.
>>>>>>>
>>>>>>> yarn.scheduler.maximum-allocation-vcores will be configured in
>>>>>>> resource manager and hence will be common for a cluster which
can possibly
>>>>>>> have multiple nodes with heterogeneous resource capabilities
>>>>>>>
>>>>>>> yarn.nodemanager.resource.cpu-vcores on the other hand will have
to
>>>>>>> be configured as per resource capability of that particular node.
>>>>>>>
>>>>>>> Recently there has been work done to automatically get memory
and
>>>>>>> CPU information from underlying OS(supported OS being Linux and
Windows) if
>>>>>>> configured to do so. This change would be available in 2.8
>>>>>>> I hope this answers your question.
>>>>>>>
>>>>>>> Regards,
>>>>>>> Varun Saxena.
>>>>>>>
>>>>>>> On Sun, Aug 23, 2015 at 9:40 PM, Pedro Magalhaes <
>>>>>>> pedrorjbr@gmail.com> wrote:
>>>>>>>
>>>>>>>> I was looking at default parameters for:
>>>>>>>>
>>>>>>>> yarn.nodemanager.resource.cpu-vcores = 8
>>>>>>>> yarn.scheduler.maximum-allocation-vcores = 32
>>>>>>>>
>>>>>>>> For me this two parameters as default doesnt make any sense.
>>>>>>>>
>>>>>>>> The first one say "the number of CPU cores that can be allocated
>>>>>>>> for containers." (I imagine that is vcore) The seconds says:
"The maximum
>>>>>>>> allocation for every container request at the RM". In my
opinion, the
>>>>>>>> second one must be equal or less than the first one.
>>>>>>>>
>>>>>>>> How can allocate 32 vcores for a container if i have only
8 cores
>>>>>>>> available per container?
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message