Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Thu, 2 Feb 2017 11:13:55 +0000 (UTC)
From: "Sunil G (JIRA)" <jira@apache.org>
To: yarn-issues@hadoop.apache.org
Message-ID: <JIRA.13021186.1479315855000.16187.1486034035024@Atlassian.JIRA>
In-Reply-To: <JIRA.13021186.1479315855000@Atlassian.JIRA>
References: <JIRA.13021186.1479315855000@Atlassian.JIRA> <JIRA.13021186.1479315855295@jira-lw-us.apache.org>
Subject: [jira] [Updated] (YARN-5889) Improve user-limit calculation in
 capacity scheduler
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Thu, 02 Feb 2017 11:14:16 -0000


     [ https://issues.apache.org/jira/browse/YARN-5889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sunil G updated YARN-5889:
--------------------------
    Attachment: YARN-5889.0009.patch

Thanks [~leftnoteasy] for helping to review the patch thoroughly. Updating a new patch

bq.4) isRecomputeNeeded:
I am slightly confused here. I think we might need null check. I ll help to share detailed view for that.

Assume that there are no precomputed user-limit at the start when RM is started or queue is refreshed. So all cache ill be empty, and we ll do our first computation when a container request comes. 
So in this case, userLimitPerSchedulingMode will be null. And we ll do a recompute and then userLimitPerSchedulingMode will have some entires. So a null check is needed at the very beginning scenario. I can see whether this check can be done outside or note. Am i missing something here? pls help to share your view.

bq.And also, we don't need latestVersionOfUserCount, instead we should call latestVersionOfUsersState.get().
userLimitNeedsRecompute or getLatestVersionOfUsersState are not writeLock protected. Hence in getComputedResourceLimitForAll/ActiveUsers  , it may be possible that latestVersionOfUsersState may change within writeLock block while operating.  Since we save the saved version of latestVersionOfUserCount to update local cache (per partition nd sch mode), even though some other thread changed the real  latestVersionOfUsersState, cache will be invalidate it correctly. Pls pool in your thoughts.

> Improve user-limit calculation in capacity scheduler
> ----------------------------------------------------
>
>                 Key: YARN-5889
>                 URL: https://issues.apache.org/jira/browse/YARN-5889
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: capacity scheduler
>            Reporter: Sunil G
>            Assignee: Sunil G
>         Attachments: YARN-5889.0001.patch, YARN-5889.0001.suggested.patchnotes, YARN-5889.0002.patch, YARN-5889.0003.patch, YARN-5889.0004.patch, YARN-5889.0005.patch, YARN-5889.0006.patch, YARN-5889.0007.patch, YARN-5889.0008.patch, YARN-5889.0009.patch, YARN-5889.v0.patch, YARN-5889.v1.patch, YARN-5889.v2.patch
>
>
> Currently user-limit is computed during every heartbeat allocation cycle with a write lock. To improve performance, this tickets is focussing on moving user-limit calculation out of heartbeat allocation flow.


--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org