hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sunil G (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (YARN-5889) Improve user-limit calculation in capacity scheduler
Date Thu, 02 Feb 2017 11:13:55 GMT

     [ https://issues.apache.org/jira/browse/YARN-5889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Sunil G updated YARN-5889:
    Attachment: YARN-5889.0009.patch

Thanks [~leftnoteasy] for helping to review the patch thoroughly. Updating a new patch

bq.4) isRecomputeNeeded:
I am slightly confused here. I think we might need null check. I ll help to share detailed
view for that.

Assume that there are no precomputed user-limit at the start when RM is started or queue is
refreshed. So all cache ill be empty, and we ll do our first computation when a container
request comes. 
So in this case, userLimitPerSchedulingMode will be null. And we ll do a recompute and then
userLimitPerSchedulingMode will have some entires. So a null check is needed at the very beginning
scenario. I can see whether this check can be done outside or note. Am i missing something
here? pls help to share your view.

bq.And also, we don't need latestVersionOfUserCount, instead we should call latestVersionOfUsersState.get().
userLimitNeedsRecompute or getLatestVersionOfUsersState are not writeLock protected. Hence
in getComputedResourceLimitForAll/ActiveUsers  , it may be possible that latestVersionOfUsersState
may change within writeLock block while operating.  Since we save the saved version of latestVersionOfUserCount
to update local cache (per partition nd sch mode), even though some other thread changed the
real  latestVersionOfUsersState, cache will be invalidate it correctly. Pls pool in your thoughts.

> Improve user-limit calculation in capacity scheduler
> ----------------------------------------------------
>                 Key: YARN-5889
>                 URL: https://issues.apache.org/jira/browse/YARN-5889
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: capacity scheduler
>            Reporter: Sunil G
>            Assignee: Sunil G
>         Attachments: YARN-5889.0001.patch, YARN-5889.0001.suggested.patchnotes, YARN-5889.0002.patch,
YARN-5889.0003.patch, YARN-5889.0004.patch, YARN-5889.0005.patch, YARN-5889.0006.patch, YARN-5889.0007.patch,
YARN-5889.0008.patch, YARN-5889.0009.patch, YARN-5889.v0.patch, YARN-5889.v1.patch, YARN-5889.v2.patch
> Currently user-limit is computed during every heartbeat allocation cycle with a write
lock. To improve performance, this tickets is focussing on moving user-limit calculation out
of heartbeat allocation flow.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message