hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jeremy p <athomewithagroove...@gmail.com>
Subject Capacity Scheduler question
Date Fri, 22 Mar 2013 21:48:02 GMT
I have two jobs, Job A and Job B.  Job A needs to run with 18 mappers per
machine, Job B needs to run with 1 mapper per machine.  Hadoop doesn't give
you a way to specify number of mappers on a per-job basis.
mapred.tasktracker.map.tasks.maximum and mapred.map.tasks do absolutely
nothing.  I've been looking into the Capacity Scheduler, but I'm unsure if
it can help me.  In
this<http://hadoop.apache.org/docs/stable/capacity_scheduler.html>documentation,
all the settings under "Resource Allocation" are
cluster-wide.  I need to be able to set the maximum capacity on a given
machine.  It does look like you have the option to set the required amount
of memory per slot, but that setting applies to all the queues.  If I could
set that value on a per-queue basis, that would be helpful.

Will the capacity scheduler help me here?  Or am I barking up the wrong
tree?  If the capacity scheduler won't help me, can you think of anything
that will?

Thanks!

--Jeremy

Mime
View raw message