hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Allen Wittenauer (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-1998) Size-based queuing for capacity scheduler
Date Fri, 06 Aug 2010 19:37:16 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-1998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12896134#action_12896134

Allen Wittenauer commented on MAPREDUCE-1998:

So the way I imagine this working is:

admin configures 3 queues:

jobs with > 1000 tasks
jobs with <999 >100 tasks
jobs with < 99 tasks

user submits a job to default, which triggers sized-based queuing.

hadoop determines map tasks: lets say 100, and user requested 20 reduces.  Total task count:

JobTracker places job in the middle tier queue since there are 120 tasks.

> Size-based queuing for capacity scheduler
> -----------------------------------------
>                 Key: MAPREDUCE-1998
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1998
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>          Components: contrib/capacity-sched
>            Reporter: Allen Wittenauer
> On job submission, it would be useful if the capacity scheduler could pick a queue based
on the # of maps and reduces.  This way one could have queues based on job-size without users
having to pick the queue prior to submission.  

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message