hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arun Suresh (JIRA)" <j...@apache.org>
Subject [jira] [Created] (YARN-8827) Plumb per app, per user and per queue resource utilization from the NM to RM
Date Thu, 27 Sep 2018 02:52:00 GMT
Arun Suresh created YARN-8827:

             Summary: Plumb per app, per user and per queue resource utilization from the
NM to RM
                 Key: YARN-8827
                 URL: https://issues.apache.org/jira/browse/YARN-8827
             Project: Hadoop YARN
          Issue Type: Sub-task
            Reporter: Arun Suresh
            Assignee: Arun Suresh

Opportunistic Containers for OverAllocation need to be allocated to pending applications in
some fair manner. Rather than evaluating queue and user resource usage (allocated resource
usage) and comparing against queue and user limits to decide the allocation, it might be make
more sense to use a snapshot of actual resource utilization of the queue and user.

To facilitate this, this JIRA proposes to aggregate per user, per app (and maybe per queue)
resource utilization in addition to aggregated Container and Node Utilization and send it
along with the NM heartbeat. It should be fairly inexpensive to aggregate - since it can be
performed in the same loop of the {{ContainersMonitorImpl}}'s Monitoring thread.

A snapshot aggregate can be made every couple of seconds in the RM. This instantaneous resource
utilization should be used to decide if Opportunistic containers can be allocated to an App,
Queue or User.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message