hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Íñigo Goiri (JIRA) <j...@apache.org>
Subject [jira] [Commented] (YARN-8827) Plumb per app, per user and per queue resource utilization from the NM to RM
Date Mon, 08 Oct 2018 17:03:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-8827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16642148#comment-16642148

Íñigo Goiri commented on YARN-8827:

Thanks [~asuresh] for the patch.
I think we can fix the findbugs, checkstyles and so on from Yetus.

Other minor comments:
* I think that ContainersMonitorImpl#run is single threaded and all the accesses to {{perAppResourceUtilization}}
are made here, no need for the thread-safe comment; nothing is here.
* In ContainersMonitorImpl, we are doing the change of units twice (#825), can we refactor
that into a proper ResourceUtilization and use it in both places?
* ContainersMonitorImpl#1106, what does it mean?
* TestResourceUtilizationAggregator#line66 fits in one single line. (There are a few more
places like #171, #174, etc).
* TestResourceUtilizationAggregator for values like 8192, you can do 8*GB as the rest of the
* Can we do a more informed wait in TestResourceUtilizationAggregator instead of sleep(1000)?
* mkmap() and e() are an OK pattern but I would probably call the second one mkmapentry and
add javadoc for the two explaining the purpose.
* Can we use an static import for {{Assert.assertEquals()}}?
* The code in {{testResourceUtilizationAggregation}} is a little repetitive. Given that this
class is only used for this, I would make the NMs part of the class and trigger registrations
and so on as part of the class. I would also add a couple more comments throughout explaining
how the apps are submitting requests, etc. Specially the numbers after #184.
* TestResourceUtilizationAggregator#184 can it be made calculated from some assignment we

> Plumb per app, per user and per queue resource utilization from the NM to RM
> ----------------------------------------------------------------------------
>                 Key: YARN-8827
>                 URL: https://issues.apache.org/jira/browse/YARN-8827
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>            Reporter: Arun Suresh
>            Assignee: Arun Suresh
>            Priority: Major
>         Attachments: YARN-8827-YARN-1011.01.patch
> Opportunistic Containers for OverAllocation need to be allocated to pending applications
in some fair manner. Rather than evaluating queue and user resource usage (allocated resource
usage) and comparing against queue and user limits to decide the allocation, it might  make
more sense to use a snapshot of actual resource utilization of the queue and user.
> To facilitate this, this JIRA proposes to aggregate per user, per app (and maybe per
queue) resource utilization in addition to aggregated Container and Node Utilization and send
it along with the NM heartbeat. It should be fairly inexpensive to aggregate - since it can
be performed in the same loop of the {{ContainersMonitorImpl}}'s Monitoring thread.
> A snapshot aggregate can be made every couple of seconds in the RM. This instantaneous
resource utilization should be used to decide if Opportunistic containers can be allocated
to an App, Queue or User.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message