hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jeremy p <athomewithagroove...@gmail.com>
Subject Re: Capacity Scheduler question
Date Fri, 22 Mar 2013 22:30:12 GMT
Thanks for the help.  Sadly, I don't think the Fair Scheduler will help me
here.  It will let you specify the number of
concurrent task slots for a pool, but that applies to the entire cluster.
 For a given pool, I need to set the maximum number of task slots per
machine.

On Fri, Mar 22, 2013 at 3:06 PM, Serge Blazhievsky <hadoop.ca@gmail.com>wrote:

> Take a look at fair scheduler it will do what you ask for
>
> Sent from my iPhone
>
> On Mar 22, 2013, at 2:48 PM, jeremy p <athomewithagroovebox@gmail.com>
> wrote:
>
> I have two jobs, Job A and Job B.  Job A needs to run with 18 mappers per
> machine, Job B needs to run with 1 mapper per machine.  Hadoop doesn't give
> you a way to specify number of mappers on a per-job basis.
> mapred.tasktracker.map.tasks.maximum and mapred.map.tasks do absolutely
> nothing.  I've been looking into the Capacity Scheduler, but I'm unsure if
> it can help me.  In this<http://hadoop.apache.org/docs/stable/capacity_scheduler.html>documentation,
all the settings under "Resource Allocation" are
> cluster-wide.  I need to be able to set the maximum capacity on a given
> machine.  It does look like you have the option to set the required amount
> of memory per slot, but that setting applies to all the queues.  If I could
> set that value on a per-queue basis, that would be helpful.
>
> Will the capacity scheduler help me here?  Or am I barking up the wrong
> tree?  If the capacity scheduler won't help me, can you think of anything
> that will?
>
> Thanks!
>
> --Jeremy
>
>

Mime
View raw message