hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Haibo Chen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-7064) Use cgroup to get container resource utilization
Date Thu, 11 Jan 2018 22:45:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-7064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16323135#comment-16323135

Haibo Chen commented on YARN-7064:

Thanks [~miklos.szegedi@cloudera.com] for the update! A few more comments on the new patch:

1) cgroupsLogged and cgroupsErrorLogged in ContainersMonitorImpl are no longer used, thus
can be removed.  
2) CombinedResourceCalculator.initialize() should probably just call `cgroup.initialize()
and procfs.initialize()` for easy maintenance. Can we call cgroup.getProcessTreeDump() in
CombinedResourceCalculator.getProcessTreeDump() instead of returning null?
3) In CgroupsResourceCalculator, how about we give more information in initialize() when CGroupsResourceCalculator
is not available to tells user what is required, like `CGroupsResourceCalculator is only available
on Linux when cgroup memory and cpu is turned on`? In updateProcessTree() and getMemorySize(),
I think not catching the YarnException would be more appropriate. The exception, if not caught
in updateProcessTree() and getMemorySize(), will be eventually caught and logged in COntainersMonitorImpl
which makes the error message easier to understand. Swallowing the exception in updateProcessTree()
and getMemorySize() will lead old (for cpu usage) or wrong (for memory) number to be reported
to ContainersMonitor, which is harder to debug. 

I will try the patch in a cluster in the meantime.

> Use cgroup to get container resource utilization
> ------------------------------------------------
>                 Key: YARN-7064
>                 URL: https://issues.apache.org/jira/browse/YARN-7064
>             Project: Hadoop YARN
>          Issue Type: Improvement
>            Reporter: Miklos Szegedi
>            Assignee: Miklos Szegedi
>         Attachments: YARN-7064.000.patch, YARN-7064.001.patch, YARN-7064.002.patch, YARN-7064.003.patch,
YARN-7064.004.patch, YARN-7064.005.patch, YARN-7064.007.patch, YARN-7064.008.patch, YARN-7064.009.patch,
> This is an addendum to YARN-6668. What happens is that that jira always wants to rebase
patches against YARN-1011 instead of trunk.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message