hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Serge Blazhievsky <hadoop...@gmail.com>
Subject Re: Capacity Scheduler question
Date Fri, 22 Mar 2013 22:06:49 GMT
Take a look at fair scheduler it will do what you ask for

Sent from my iPhone

On Mar 22, 2013, at 2:48 PM, jeremy p <athomewithagroovebox@gmail.com> wrote:

> I have two jobs, Job A and Job B.  Job A needs to run with 18 mappers per machine, Job
B needs to run with 1 mapper per machine.  Hadoop doesn't give you a way to specify number
of mappers on a per-job basis.  mapred.tasktracker.map.tasks.maximum and mapred.map.tasks
do absolutely nothing.  I've been looking into the Capacity Scheduler, but I'm unsure if it
can help me.  In this documentation, all the settings under "Resource Allocation" are cluster-wide.
 I need to be able to set the maximum capacity on a given machine.  It does look like you
have the option to set the required amount of memory per slot, but that setting applies to
all the queues.  If I could set that value on a per-queue basis, that would be helpful.
> Will the capacity scheduler help me here?  Or am I barking up the wrong tree?  If the
capacity scheduler won't help me, can you think of anything that will?
> Thanks!
> --Jeremy

View raw message