hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3243) CapacityScheduler should pass headroom from parent to children to make sure ParentQueue obey its capacity limits.
Date Thu, 12 Mar 2015 19:37:39 GMT

    [ https://issues.apache.org/jira/browse/YARN-3243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14359230#comment-14359230
] 

Hadoop QA commented on YARN-3243:
---------------------------------

{color:red}-1 overall{color}.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12704207/YARN-3243.5.patch
  against trunk revision 06ce1d9.

    {color:green}+1 @author{color}.  The patch does not contain any @author tags.

    {color:red}-1 tests included{color}.  The patch doesn't appear to include any new or modified
tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    {color:green}+1 javac{color}.  The applied patch does not increase the total number of
javac compiler warnings.

    {color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

    {color:green}+1 eclipse:eclipse{color}.  The patch built with eclipse:eclipse.

    {color:red}-1 findbugs{color}.  The patch appears to introduce 5 new Findbugs (version
2.0.3) warnings.

    {color:green}+1 release audit{color}.  The applied patch does not increase the total number
of release audit warnings.

    {color:red}-1 core tests{color}.  The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:

                  org.apache.hadoop.yarn.server.resourcemanager.security.TestAMRMTokens
                  org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestAllocationFileLoaderService
                  org.apache.hadoop.yarn.server.resourcemanager.TestKillApplicationWithRMHA
                  org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerQueueACLs
                  org.apache.hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStoreZKClientConnections
                  org.apache.hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStore
                  org.apache.hadoop.yarn.server.resourcemanager.recovery.TestLeveldbRMStateStore
                  org.apache.hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStorePerf
                  org.apache.hadoop.yarn.server.resourcemanager.scheduler.TestSchedulerUtils
                  org.apache.hadoop.yarn.server.resourcemanager.security.TestClientToAMTokens
                  org.apache.hadoop.yarn.server.resourcemanager.recovery.TestFSRMStateStore
                  org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService
                  org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization

Test results: https://builds.apache.org/job/PreCommit-YARN-Build/6942//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-YARN-Build/6942//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6942//console

This message is automatically generated.

> CapacityScheduler should pass headroom from parent to children to make sure ParentQueue
obey its capacity limits.
> -----------------------------------------------------------------------------------------------------------------
>
>                 Key: YARN-3243
>                 URL: https://issues.apache.org/jira/browse/YARN-3243
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: capacityscheduler, resourcemanager
>            Reporter: Wangda Tan
>            Assignee: Wangda Tan
>         Attachments: YARN-3243.1.patch, YARN-3243.2.patch, YARN-3243.3.patch, YARN-3243.4.patch,
YARN-3243.5.patch
>
>
> Now CapacityScheduler has some issues to make sure ParentQueue always obeys its capacity
limits, for example:
> 1) When allocating container of a parent queue, it will only check parentQueue.usage
< parentQueue.max. If leaf queue allocated a container.size > (parentQueue.max - parentQueue.usage),
parent queue can excess its max resource limit, as following example:
> {code}
>         A  (usage=54, max=55)
>        /     \
>       A1     A2 (usage=1, max=55)
> (usage=53, max=53)
> {code}
> Queue-A2 is able to allocate container since its usage < max, but if we do that, A's
usage can excess A.max.
> 2) When doing continous reservation check, parent queue will only tell children "you
need unreserve *some* resource, so that I will less than my maximum resource", but it will
not tell how many resource need to be unreserved. This may lead to parent queue excesses configured
maximum capacity as well.
> With YARN-3099/YARN-3124, now we have {{ResourceUsage}} class in each class, *here is
my proposal*:
> - ParentQueue will set its children's ResourceUsage.headroom, which means, *maximum resource
its children can allocate*.
> - ParentQueue will set its children's headroom to be (saying parent's name is "qA"):
min(qA.headroom, qA.max - qA.used). This will make sure qA's ancestors' capacity will be enforced
as well (qA.headroom is set by qA's parent).
> - {{needToUnReserve}} is not necessary, instead, children can get how much resource need
to be unreserved to keep its parent's resource limit.
> - More over, with this, YARN-3026 will make a clear boundary between LeafQueue and FiCaSchedulerApp,
headroom will consider user-limit, etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message