hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Payne (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (YARN-5889) Improve user-limit calculation in capacity scheduler
Date Sat, 07 Jan 2017 17:23:58 GMT

     [ https://issues.apache.org/jira/browse/YARN-5889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Eric Payne updated YARN-5889:
    Attachment: YARN-5889.0001.suggested.patchnotes

Thanks [~sunilg] for providing this work.

I am concerned about the way in which {{LeafQueue#getComputedUserLimit}} and {{LeafQueue#getComputedActiveUserLimit}}
are deciding when to recompute user limits.
  - In the case of {{getComputedActiveUserLimit}}, if number of active users has increased
or decreased, all active users in {{preComputedActiveUserLimit}} are invalided, and not just
the one that was activated/deactivated. This requires recalculation for other users when it
is not necessary.
  - In the case of {{getComputedUserLimit}}, comparing a cached {{lastUserCount}} with {{users.size()}}
could miss occasions when recomputation would be necessary.

My understanding of [~jlowe]'s [comment (above)|https://issues.apache.org/jira/browse/YARN-5889?focusedCommentId=15745552&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15745552]
is that the {{LeafQueue}} would increment a change counter which would be cached in each {{User}}'s
object and then would be used to determine when recalculation of user limit should occur.

Rather than document each occurrence, I have uploaded a patch with my suggestions. {{YARN-5889.0001.suggested.patchnotes}}
is the diff between {{YARN-5889.0001.patch}} and my suggestions.

I haven't tested all of the scenarios yet (I'll complete that soon), but I think this covers
my concerns about the current patch, and I think it is more efficient. Please have a look
and let me know your thoughts.

> Improve user-limit calculation in capacity scheduler
> ----------------------------------------------------
>                 Key: YARN-5889
>                 URL: https://issues.apache.org/jira/browse/YARN-5889
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: capacity scheduler
>            Reporter: Sunil G
>            Assignee: Sunil G
>         Attachments: YARN-5889.0001.patch, YARN-5889.0001.suggested.patchnotes, YARN-5889.v0.patch,
YARN-5889.v1.patch, YARN-5889.v2.patch
> Currently user-limit is computed during every heartbeat allocation cycle with a write
lock. To improve performance, this tickets is focussing on moving user-limit calculation out
of heartbeat allocation flow.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message