hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Íñigo Goiri (JIRA) <j...@apache.org>
Subject [jira] [Commented] (YARN-8827) Plumb per app, per user and per queue resource utilization from the NM to RM
Date Tue, 09 Oct 2018 16:42:01 GMT

    [ https://issues.apache.org/jira/browse/YARN-8827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16643713#comment-16643713
] 

Íñigo Goiri commented on YARN-8827:
-----------------------------------

Some minor comments:
* NodeStatusPBImpl extra line before addAppUtilizations.
* NodeStatusPBImpl#63 fits in one line.
* In NodeStatusPBImpl#128, I would extract the key and the value for readability (e.g., appId
= entry.getKey()).
* Similar for initAppUtilizations.
* Can we use logger in ResourceUtilizationAggregator and use the {} pattern for the logs?
* We do a lot the pattern of new empty utilization, I think we could create ResourceUtilization.newInstance()
setting all to 0, or newZero(). Actually the pattern for utilization maps is always the same.
At some point it might be useful to have a UtilizationMap.
* In MockNM can we keep the old method passing null by default for app utilization and just
add the new method instead of replacing?
* We can resolve a few of the remaining checkstyle, I think it is fine to make the fields
private for example.

Other than this minor things, this LGTM.
Whenever we open the follow-up JIRA that uses this, can you link it to this one?

> Plumb per app, per user and per queue resource utilization from the NM to RM
> ----------------------------------------------------------------------------
>
>                 Key: YARN-8827
>                 URL: https://issues.apache.org/jira/browse/YARN-8827
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>            Reporter: Arun Suresh
>            Assignee: Arun Suresh
>            Priority: Major
>         Attachments: YARN-8827-YARN-1011.01.patch, YARN-8827-YARN-1011.02.patch, YARN-8827-YARN-1011.03.patch,
YARN-8827-YARN-1011.04.patch, YARN-8827-YARN-1011.05.patch
>
>
> Opportunistic Containers for OverAllocation need to be allocated to pending applications
in some fair manner. Rather than evaluating queue and user resource usage (allocated resource
usage) and comparing against queue and user limits to decide the allocation, it might  make
more sense to use a snapshot of actual resource utilization of the queue and user.
> To facilitate this, this JIRA proposes to aggregate per user, per app (and maybe per
queue) resource utilization in addition to aggregated Container and Node Utilization and send
it along with the NM heartbeat. It should be fairly inexpensive to aggregate - since it can
be performed in the same loop of the {{ContainersMonitorImpl}}'s Monitoring thread.
> A snapshot aggregate can be made every couple of seconds in the RM. This instantaneous
resource utilization should be used to decide if Opportunistic containers can be allocated
to an App, Queue or User.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message