hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Lowe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5356) NodeManager should communicate physical resource capability to ResourceManager
Date Thu, 03 Nov 2016 14:34:58 GMT

    [ https://issues.apache.org/jira/browse/YARN-5356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15632892#comment-15632892

Jason Lowe commented on YARN-5356:

Thanks for the patch, [~elgoiri]!  Looks good overall, but I wonder about this change:
    // Get actual node physical resources
    int physicalMemoryMb = memoryMb;
    int physicalCores = virtualCores;
    ResourceCalculatorPlugin rcp =
        ResourceCalculatorPlugin.getResourceCalculatorPlugin(null, conf);
    if (rcp != null) {
      physicalMemoryMb = (int) (rcp.getPhysicalMemorySize() / (1024 * 1024));
      physicalCores = rcp.getNumProcessors();
    this.physicalResource =
        Resource.newInstance(physicalMemoryMb, physicalCores);

This is not using the resource calculator the user may have configured for yarn.nodemanager.resource-calculator.class,
and IIUC the value being reported here for total physical node resources will be compared
against values being returned by that configured class.  If the two classes don't match up
then there could be some "interesting" and unexpected results.  I think we need to be consistent
and use the same class that's being used for resource monitoring.

Rather than using read/write locks, we could avoid locks altogether by declaring the RMNodeImpl
physicalResource field as volatile.

> NodeManager should communicate physical resource capability to ResourceManager
> ------------------------------------------------------------------------------
>                 Key: YARN-5356
>                 URL: https://issues.apache.org/jira/browse/YARN-5356
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: nodemanager, resourcemanager
>    Affects Versions: 3.0.0-alpha1
>            Reporter: Nathan Roberts
>            Assignee: Inigo Goiri
>              Labels: oct16-medium
>         Attachments: YARN-5356.000.patch, YARN-5356.001.patch, YARN-5356.002.patch, YARN-5356.002.patch,
YARN-5356.003.patch, YARN-5356.004.patch, YARN-5356.005.patch, YARN-5356.006.patch, YARN-5356.007.patch
> Currently ResourceUtilization contains absolute quantities of resource used (e.g. 4096MB
memory used). It would be good if the NM also communicated the actual physical resource capabilities
of the node so that the RM can use this data to schedule more effectively (overcommit, etc)
> Currently the only available information is the Resource the node registered with (or
later updated using updateNodeResource). However, these aren't really sufficient to get a
good view of how utilized a resource is. For example, if a node reports 400% CPU utilization,
does that mean it's completely full, or barely utilized? Today there is no reliable way to
figure this out.
> [~elgoiri] - Lots of good work is happening in YARN-2965 so curious if you have thoughts/opinions
on this?

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message