hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Lowe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5356) NodeManager should communicate physical resource capability to ResourceManager
Date Fri, 04 Nov 2016 20:37:58 GMT

    [ https://issues.apache.org/jira/browse/YARN-5356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15637606#comment-15637606

Jason Lowe commented on YARN-5356:

Thanks for updating the patch!

I don't think getNMResourceCalculatorPlugin does quite the right thing for this case.  Note
that it tries to get the configured container monitor resource calculator with a fallback
to the node monitor resource calculator.  The resource monitor only uses the node monitor
resource calculator, not the container monitor resource calculator.  So theoretically we could
be doing the wrong thing if the container monitor calculator did something different than
the node monitor calculator.  We need to use the same calculator that the NodeResourceMonitorImpl
does, otherwise we can still have the mismatched calculators issue.

We could add two utility methods for creating the calculator, one for the node monitor and
one for the container monitor, and make sure we call the same one that NodeResourceMonitorImpl

> NodeManager should communicate physical resource capability to ResourceManager
> ------------------------------------------------------------------------------
>                 Key: YARN-5356
>                 URL: https://issues.apache.org/jira/browse/YARN-5356
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: nodemanager, resourcemanager
>    Affects Versions: 3.0.0-alpha1
>            Reporter: Nathan Roberts
>            Assignee: Inigo Goiri
>              Labels: oct16-medium
>         Attachments: YARN-5356.000.patch, YARN-5356.001.patch, YARN-5356.002.patch, YARN-5356.002.patch,
YARN-5356.003.patch, YARN-5356.004.patch, YARN-5356.005.patch, YARN-5356.006.patch, YARN-5356.007.patch,
YARN-5356.008.patch, YARN-5356.009.patch, YARN-5356.010.patch
> Currently ResourceUtilization contains absolute quantities of resource used (e.g. 4096MB
memory used). It would be good if the NM also communicated the actual physical resource capabilities
of the node so that the RM can use this data to schedule more effectively (overcommit, etc)
> Currently the only available information is the Resource the node registered with (or
later updated using updateNodeResource). However, these aren't really sufficient to get a
good view of how utilized a resource is. For example, if a node reports 400% CPU utilization,
does that mean it's completely full, or barely utilized? Today there is no reliable way to
figure this out.
> [~elgoiri] - Lots of good work is happening in YARN-2965 so curious if you have thoughts/opinions
on this?

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message