hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sergey Tryuber (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-3859) CapacityScheduler incorrectly utilizes extra-resources of queue for high-memory jobs
Date Thu, 16 May 2013 06:15:17 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-3859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13659278#comment-13659278
] 

Sergey Tryuber commented on MAPREDUCE-3859:
-------------------------------------------

Mike, that's great! So I think this task can be closed, unless someone from Cloudera (their
MR1 in CDH4 is still be affected) wants to take care about this issue and port the fix to
old Capacity Scheduler into their sources.

For the others who faces this issue, below is a brief step-by-step instruction for CDH4.1.2:
   
* Download sources from https://ccp.cloudera.com/display/SUPPORT/CDH+Downloads. Note: you
need hadoop-0.20-mapreduce-0.20.2+1265 tarball.
* Unpack it and go to root directory.
* Apply changes from the first comment and test case from attached patch. 
* Also you should add the following lines:
{code}
    reactor.repo=https\://repository.cloudera.com/content/repositories/snapshots
    version=2.0.0-mr1-cdh4.1.2
{code}
    into src/contrib/index/ivy/libraries.properties and src/contrib/capacity-scheduler/ivy/libraries.properties
files.
* Test fixes that were made:
{code}
    ant test-contrib
{code}
* Build a jar file:
{code}
    cd src/contrib/capacity-scheduler/
    ant jar
    cd -
{code}
* The result file will be placed at build/contrib/capacity-scheduler/hadoop-capacity-scheduler-2.0.0-mr1-cdh4.1.2.jar.
* Replace original file with the fixed on a node where JobTracker is started. Original file
is placed in /usr/lib/hadoop-0.20-mapreduce/contrib/capacity-scheduler/ directory.
                
> CapacityScheduler incorrectly utilizes extra-resources of queue for high-memory jobs
> ------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-3859
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3859
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: capacity-sched
>    Affects Versions: 1.0.0
>         Environment: CDH3u1
>            Reporter: Sergey Tryuber
>            Assignee: Sergey Tryuber
>         Attachments: test-to-fail.patch.txt
>
>
> Imagine, we have a queue A with capacity 10 slots and 20 as extra-capacity, jobs which
use 3 map slots will never consume more than 9 slots, regardless how many free slots on a
cluster.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message