hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jeremy p <athomewithagroove...@gmail.com>
Subject Capacity Scheduler question
Date Fri, 22 Mar 2013 21:48:02 GMT
I have two jobs, Job A and Job B.  Job A needs to run with 18 mappers per
machine, Job B needs to run with 1 mapper per machine.  Hadoop doesn't give
you a way to specify number of mappers on a per-job basis.
mapred.tasktracker.map.tasks.maximum and mapred.map.tasks do absolutely
nothing.  I've been looking into the Capacity Scheduler, but I'm unsure if
it can help me.  In
all the settings under "Resource Allocation" are
cluster-wide.  I need to be able to set the maximum capacity on a given
machine.  It does look like you have the option to set the required amount
of memory per slot, but that setting applies to all the queues.  If I could
set that value on a per-queue basis, that would be helpful.

Will the capacity scheduler help me here?  Or am I barking up the wrong
tree?  If the capacity scheduler won't help me, can you think of anything
that will?



View raw message