hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thomas Graves (JIRA)" <j...@apache.org>
Subject [jira] [Created] (MAPREDUCE-4191) capacity scheduler: job unexpectedly exceeds queue capacity limit by one task
Date Mon, 23 Apr 2012 17:38:33 GMT
Thomas Graves created MAPREDUCE-4191:
----------------------------------------

             Summary: capacity scheduler: job unexpectedly exceeds queue capacity limit by
one task
                 Key: MAPREDUCE-4191
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4191
             Project: Hadoop Map/Reduce
          Issue Type: Bug
          Components: mrv2, scheduler
    Affects Versions: 0.23.3
            Reporter: Thomas Graves


While testing the queue capacity limits, it appears that the job can exceed the
queue capacity limit by one task while the user limit factor is 1. It's not
clear to me why this is. 

Here is the steps to reproduce:

1) set yarn.app.mapreduce.am.resource.mb to 2048 (default value)
2) set yarn.scheduler.capacity.root.default.user-limit-factor to 1.0 (default)
3) set yarn.scheduler.capacity.root.default.capacity to 90 (%)
4) For a cluster with capacity of 56G, 90% rounded up is 51.
5) submit a job with large number of tasks, each task using 1G memory. 
6) webui shows that the used resource is 52 G, which is 92.9% of the cluster
capacity (instead of the expected 90%), and 103.2% of the queue capacity
(instead of the expected 100%).




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message