hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Inigo Goiri (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5356) ResourceUtilization should also include resource availability
Date Tue, 12 Jul 2016 00:14:11 GMT

    [ https://issues.apache.org/jira/browse/YARN-5356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15371947#comment-15371947
] 

Inigo Goiri commented on YARN-5356:
-----------------------------------

In general, we have 3 values:
* Actual resources of the full machine. This currently comes from {{NodeManagerHardwareUtils}}
if I remember correctly. For example, it can be 12 cores for example
* Resource available for the Node Manager. This is currently defined in yarn-site.xml with
key {{yarn.nodemanager.resource.cpu-vcores}} or with the {{updateNodeResource()}}. For example,
6 cores.
* Actual utilization of the machine. This is extracted in the {{NodeResourceMonitor}} with
the {{ResourceCalculatorPlugin}}. And it can be 400%, which would imply 4 out of the 12 cores
used.

[~nroberts], I understand that your problem is that with the current approach you know that
you have 6 cores available to the NM and 4 of them are used. However, the machine is not that
utilized (~30%). Correct? In that case, we would only need to report the actual size of the
machine at registration time as it would never change. Not sure that {{ResourceUtilization}}
would be the right place for that as it would be reported in every heartbeat continuously.


> ResourceUtilization should also include resource availability
> -------------------------------------------------------------
>
>                 Key: YARN-5356
>                 URL: https://issues.apache.org/jira/browse/YARN-5356
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: nodemanager, resourcemanager
>    Affects Versions: 3.0.0-alpha1
>            Reporter: Nathan Roberts
>
> Currently ResourceUtilization contains absolute quantities of resource used (e.g. 4096MB
memory used). It would be good if it also included how much of that resource is actually available
on the node so that the RM can use this data to schedule more effectively (overcommit, etc)
> Currently the only available information is the Resource the node registered with (or
later updated using updateNodeResource). However, these aren't really sufficient to get a
good view of how utilized a resource is. For example, if a node reports 400% CPU utilization,
does that mean it's completely full, or barely utilized? Today there is no reliable way to
figure this out.
> [~elgoiri] - Lots of good work is happening in YARN-2965 so curious if you have thoughts/opinions
on this?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message