hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Payne (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (YARN-5889) Improve user-limit calculation in capacity scheduler
Date Thu, 01 Dec 2016 14:29:58 GMT

    [ https://issues.apache.org/jira/browse/YARN-5889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15712097#comment-15712097
] 

Eric Payne edited comment on YARN-5889 at 12/1/16 2:29 PM:
-----------------------------------------------------------

bq. Any change in resource (allocation and release of container etc) for a given user could
set a state variable. This will set off by the computation thread if next cycle falls immediate.
Just so we're on the same page and what I'm suggesting is more clear.

I'm suggesting that we remove the computation thread ({{ComputeUserLimitAsyncThread}}), and
when {{LeafQueue#assignContainers}} or {{LeafQueue#releaseResource}} is called, we can set
the flag in {{UserToPartitionRecord}}. This flag should also be set when queues are refreshed.
{{getComputedUserLimit}} would then be called as it is in your patch, and compute user limit
resource if the flag is set.

bq. Its not ideal to ask allocation thread to hold till computation. So by seeing this state
variable, we might need to compute user-limit in same allocation thread.
I recognize that my suggested approach does the calculation during the allocation thread.
My assertion would be that
# this calculation is being done currently in the allocation thread
# my suggested approach is an optimization that would decrease (sometimes greatly) the number
of times the calculation happens
# my suggested approach would be more accurate than with a separate computation thread.


was (Author: eepayne):
bq. Any change in resource (allocation and release of container etc) for a given user could
set a state variable. This will set off by the computation thread if next cycle falls immediate.
Just so we're on the same page and what I'm suggesting is more clear.

I'm suggesting that we remove the computation thread ({{ComputeUserLimitAsyncThread}}), and
when {{LeafQueue#assignContainers}} or {{LeafQueue#releaseResource}} is called, we can set
the flag in {{UserToPartitionRecord}}. This flag should also be set within queues are refreshed.
{{getComputedUserLimit}} would then be called as it is now, and compute user limit resource
if the flag is set.

bq. Its not ideal to ask allocation thread to hold till computation. So by seeing this state
variable, we might need to compute user-limit in same allocation thread.
I recognize that my suggested approach does the calculation during the allocation thread.
My assertion would be that
# this calculation is being done currently in the allocation thread
# my suggested approach is an optimization that would decrease (sometimes greatly) the number
of times the calculation happens
# my suggested approach would be more accurate than with a separate computation thread.

> Improve user-limit calculation in capacity scheduler
> ----------------------------------------------------
>
>                 Key: YARN-5889
>                 URL: https://issues.apache.org/jira/browse/YARN-5889
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: capacity scheduler
>            Reporter: Sunil G
>            Assignee: Sunil G
>         Attachments: YARN-5889.v0.patch, YARN-5889.v1.patch, YARN-5889.v2.patch
>
>
> Currently user-limit is computed during every heartbeat allocation cycle with a write
lock. To improve performance, this tickets is focussing on moving user-limit calculation out
of heartbeat allocation flow.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message