hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Maysam Yabandeh (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2430) FairShareComparator: cache the results of getResourceUsage()
Date Tue, 19 Aug 2014 01:42:18 GMT

    [ https://issues.apache.org/jira/browse/YARN-2430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14101686#comment-14101686

Maysam Yabandeh commented on YARN-2430:

Here are the current alternative solutions:

1. a simple, quick fix would be to cache the result of getResourceUsage in a field of Schedulable
and invalidate the cache after each scheduling. The invalidation requires iteration on all
schedulables with cost O( n ).

2. alternatively as suggested by Karthik the cached result could be updated periodically as
part of UpdateThread. This approach would also encourage moving the sorting also to the UpdateThread
since the sort algorithm is no longer provided with the most up-to-date data.

3. Karthik also brought up the option of bottom-up update of the resource usage when something
gets updated: each Schedulable pushes up the change in its resource usage after each change.
This would require invoking the push-up method at the right places. Care must be taken in
future changes not to forget calling the push-up method.

I would highly appreciate the comments.

> FairShareComparator: cache the results of getResourceUsage()
> ------------------------------------------------------------
>                 Key: YARN-2430
>                 URL: https://issues.apache.org/jira/browse/YARN-2430
>             Project: Hadoop YARN
>          Issue Type: Improvement
>            Reporter: Maysam Yabandeh
>            Assignee: Maysam Yabandeh
> The compare of FairShareComparator has 3 invocation of  getResourceUsage per comparable
object. In the case of queues, the implementation of getResourceUsage requires iterating over
the apps and adding up their current usage. The compare method can reuse the result of getResourceUsage
to reduce the load by third. However, to further reduce the load the result of getResourceUsage
can be cached in FSLeafQueue. This would be more efficient since the invocation of compare
method on each Comparable object is >= 1.

This message was sent by Atlassian JIRA

View raw message