hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shane Kumpf (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-6729) NM percentage-physical-cpu-limit should be always 100 if DefaultLCEResourcesHandler is used
Date Mon, 29 Oct 2018 21:34:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-6729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16667780#comment-16667780

Shane Kumpf commented on YARN-6729:

Thanks for the patch, [~tangzhankun]! The update looks good to me overall. However, I think
these docs could use an overhaul. Below are some of my thoughts on the rework. I'm OK if we
want to open a follow on for the suggestions below, given it is a more exhaustive rework,
just let me know.


As an example, if you enable LCE and specify {{CgroupsLCEResourcesHandler}} as the handler
class, LCE has code to force the use of {{DefaultLCEResourcesHandler}}, so LCE never uses
{{CgroupsLCEResourcesHandler}}. {{CgroupsLCEResourcesHandler}} is used by the {{ResourceHandler}} though
to determine if the CPU controller should be setup, however, {{yarn.nodemanager.resource.cpu.enabled}}
does the exact same thing. Based on this, I don't think we should list {{yarn.nodemanager.linux-container-executor.resources-handler.class}}
at all in the docs and guide users towards using the {{yarn.nodemanager.resource.cpu.enabled}}

None of the other {{yarn.nodemanager.resource.*.enabled}} properties. It would be good to
have links to the elastic memory accounting as well.

Also, the table at the bottom, _CGroups and security,_ really has nothing to do with cgroups
and should probably be moved to a page about LCE.

I would look to structure these docs something like as follows:
 * Summary
 ** What is cgroups and why would a user want to enable them
 * Prerequisites for enabling YARN cgroup support
 ** Recent OS
 ** Cgroup mounts required and YARN settings
 *** Overview of YARN cgroup mounting capabilities
 ** Enabling LCE
 * Enabling YARN cgroup support
 ** CPU controller
 *** overview of pct and strict limits
 ** Memory controller
 *** Link to memory controller deep dive
 ** Disk controller
 ** Traffic shaping
 ** ...

> NM percentage-physical-cpu-limit should be always 100 if DefaultLCEResourcesHandler is
> -------------------------------------------------------------------------------------------
>                 Key: YARN-6729
>                 URL: https://issues.apache.org/jira/browse/YARN-6729
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: nodemanager
>            Reporter: Yufei Gu
>            Assignee: Zhankun Tang
>            Priority: Major
>         Attachments: YARN-6729-trunk.001.patch
> NM percentage-physical-cpu-limit is not honored in DefaultLCEResourcesHandler, which
may cause container cpu usage calculation issue. e.g. container vcore usage is potentially
more than 100% if percentage-physical-cpu-limit is set to a value less than 100. 

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message