hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2628) Capacity scheduler with DominantResourceCalculator carries out reservation even though slots are free
Date Wed, 01 Oct 2014 20:03:34 GMT

    [ https://issues.apache.org/jira/browse/YARN-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14155393#comment-14155393

Hadoop QA commented on YARN-2628:

{color:red}-1 overall{color}.  Here are the results of testing the latest attachment 
  against trunk revision 1f5b42a.

    {color:green}+1 @author{color}.  The patch does not contain any @author tags.

    {color:green}+1 tests included{color}.  The patch appears to include 1 new or modified
test files.

    {color:green}+1 javac{color}.  The applied patch does not increase the total number of
javac compiler warnings.

    {color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

    {color:green}+1 eclipse:eclipse{color}.  The patch built with eclipse:eclipse.

    {color:green}+1 findbugs{color}.  The patch does not introduce any new Findbugs (version
2.0.3) warnings.

        {color:red}-1 release audit{color}.  The applied patch generated 1 release audit warnings.

    {color:green}+1 core tests{color}.  The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

    {color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-YARN-Build/5202//testReport/
Release audit warnings: https://builds.apache.org/job/PreCommit-YARN-Build/5202//artifact/patchprocess/patchReleaseAuditProblems.txt
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5202//console

This message is automatically generated.

> Capacity scheduler with DominantResourceCalculator carries out reservation even though
slots are free
> -----------------------------------------------------------------------------------------------------
>                 Key: YARN-2628
>                 URL: https://issues.apache.org/jira/browse/YARN-2628
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: capacityscheduler
>    Affects Versions: 2.5.1
>            Reporter: Varun Vasudev
>            Assignee: Varun Vasudev
>         Attachments: apache-yarn-2628.0.patch
> We've noticed that if you run the CapacityScheduler with the DominantResourceCalculator,
sometimes apps will end up with containers in a reserved state even though free slots are
> The root cause seems to be this piece of code from CapacityScheduler.java -
> {noformat}
>     // Try to schedule more if there are no reservations to fulfill
>     if (node.getReservedContainer() == null) {
>       if (Resources.greaterThanOrEqual(calculator, getClusterResource(),
>           node.getAvailableResource(), minimumAllocation)) {
>         if (LOG.isDebugEnabled()) {
>           LOG.debug("Trying to schedule on node: " + node.getNodeName() +
>               ", available: " + node.getAvailableResource());
>         }
>         root.assignContainers(clusterResource, node, false);
>       }
>     } else {
>       LOG.info("Skipping scheduling since node " + node.getNodeID() + 
>           " is reserved by application " + 
>           node.getReservedContainer().getContainerId().getApplicationAttemptId()
>           );
>     }
> {noformat}
> The code is meant to check if a node has any slots available for containers . Since it
uses the greaterThanOrEqual function, we end up in situation where greaterThanOrEqual returns
true, even though we may not have enough CPU or memory to actually run the container.

This message was sent by Atlassian JIRA

View raw message