hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shane Kumpf (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-8924) Refine the document or code related to legacy CPU isolation/enforcement
Date Mon, 29 Oct 2018 13:48:05 GMT

    [ https://issues.apache.org/jira/browse/YARN-8924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16667223#comment-16667223

Shane Kumpf commented on YARN-8924:

Sorry for the delay. As mentioned, {{LCEResourcesHandler}} and the implementations are deprecated.
I think it makes sense to update the documentation. It looks like that is where things are
headed in YARN-6729. I'll review that issue.

> Refine the document or code related to legacy CPU isolation/enforcement
> -----------------------------------------------------------------------
>                 Key: YARN-8924
>                 URL: https://issues.apache.org/jira/browse/YARN-8924
>             Project: Hadoop YARN
>          Issue Type: Improvement
>            Reporter: Zhankun Tang
>            Assignee: Zhankun Tang
>            Priority: Minor
> This is to re-think the legacy configuration/code of CPU resource isolation. In YARN-3542,
we involve _CGroupsCpuResourceHandlerImpl_ based on new _ResourceHandler_ mechanism but
leaves the configuration "yarn.nodemanager.linux-container-executor.resources-handler.class"
there for a long time. Now it seems confusing to the end user.
> Check YARN-6729, one sets "_DefaultLCEResourcesHandler_" and found that give "percentage-physical-cpu-limit"
a value less than "100" doesn't work.
> As far as I know, internally, the _CgroupsLCEResourcesHandler_ and _DefaultLCEResourcesHandler
are_ both deprecated. YARN won't use them anymore.
> Instead, YARN uses _CGroupsCpuResourceHandlerImpl_ to do CPU isolation and only in
LCE. If we want to enforce CPU usage, we must set LCE and CgroupsLCEResourceHandler like this:
> {noformat}
> <property>
>   <description>who will execute(launch) the containers.</description>
>   <name>yarn.nodemanager.container-executor.class</name>
>   <value>org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor</value>
> </property>
> <property>
>  <description>The class which should help the LCE handle resources.</description>
>  <name>yarn.nodemanager.linux-container-executor.resources-handler.class</name>
>  <value>org.apache.hadoop.yarn.server.nodemanager.util.CgroupsLCEResourcesHandler<value>
>  </property>{noformat}
> Based on above settings can the CPU related settings like "percentage-physical-cpu-limit"
works as expected.
> To avoid confusing like YARN-6729, we can do two things:
>  # More clear document about how should user configure CPU isolation/enforcement in
>  # Make "ResourceHandlerModuler" stable and remove legacy code and update the document
to recommend new setting "yarn.nodemanager.resource.cpu.enabled"
> Thoughts? [~leftnoteasy], [~vinodkv], [~vvasudev]

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message