hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arun Suresh (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-6406) Garbage Collect unused SchedulerRequestKeys
Date Sat, 01 Apr 2017 00:13:41 GMT

    [ https://issues.apache.org/jira/browse/YARN-6406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15951845#comment-15951845

Arun Suresh commented on YARN-6406:

Thanks for the review [~leftnoteasy],

bq. Why changes of AppInfo required?
Hmmm.. so the the {{TestRMWebServicesApps}} was complaining since it needed a resource request
object. With this patch, if there is no outstanding resource request, The AppInfo will not
contain any resource request objects. So I decided to send a dummy resourceRequest Object
when none exist. I think the right approach is to fix the test case (which might be harder)..
thoughts ?

bq.  In LocalitySchedulingPlacementSet: it calls appSchedulingInfo directly in decrementOutstanding
Don't think it is a problem too much (based on existing code paths) .. but yes, maybe we should
clean it up, since it could lead some circular references to the same placementset object.
Please feel free to open another JIRA for that (I can help review), but for the timebeing,
I think we can remove the schedulerkey as is done in this patch ?

> Garbage Collect unused SchedulerRequestKeys
> -------------------------------------------
>                 Key: YARN-6406
>                 URL: https://issues.apache.org/jira/browse/YARN-6406
>             Project: Hadoop YARN
>          Issue Type: Improvement
>    Affects Versions: 2.8.0, 2.7.3, 3.0.0-alpha2
>            Reporter: Arun Suresh
>            Assignee: Arun Suresh
>         Attachments: YARN-6406.001.patch
> YARN-5540 introduced some optimizations to remove satisfied SchedulerKeys from the AppScheduleingInfo.
It looks like after YARN-6040, ScedulerRequestKeys are removed only if the Application sends
a 0 numContainers requests. While earlier, the outstanding schedulerKeys were also remove
as soon as a container is allocated as well.
> An additional optimization we were hoping to include is to remove the ResourceRequests
itself once the numContainers == 0, since we see in our clusters that the RM heap space consumption
increases drastically due to a large number of ResourceRequests with 0 num containers.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message