hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sergey Tryuber (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-3859) CapacityScheduler incorrectly utilizes extra-resources of queue for high-memory jobs
Date Thu, 16 Feb 2012 09:52:59 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-3859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13209243#comment-13209243
] 

Sergey Tryuber commented on MAPREDUCE-3859:
-------------------------------------------

Arun, Harsh, I've submitted patch (test-to-fail.patch.txt) with test (it should be applied
against "branch-1.0" branch).

Test creates 20 map slots cluster (5 nodes with 4 slots) and queue with capacity 10 slots,
max capacity 16 slots, user limit factor 2. The cluster is idle. When we submit high memory
job (5 mappers with 4 slots per mapper), it should consume 16 slots (4 mappers should be run),
but it doesn't because of the bug.

Test with fix (fix is not in the patch) works well.

Feel free to give any remarks (about code style, for example), because that's my first hadoop
coding. Arun, agree with you, that each bug report should be "double-checked".
                
> CapacityScheduler incorrectly utilizes extra-resources of queue for high-memory jobs
> ------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-3859
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3859
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: contrib/capacity-sched
>    Affects Versions: 1.0.0
>         Environment: CDH3u1
>            Reporter: Sergey Tryuber
>         Attachments: test-to-fail.patch.txt
>
>
> Imagine, we have a queue A with capacity 10 slots and 20 as extra-capacity, jobs which
use 3 map slots will never consume more than 9 slots, regardless how many free slots on a
cluster.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message