hadoop-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From james warren <ja...@rockyou.com>
Subject Re: Limiting concurrent maps
Date Wed, 20 Oct 2010 23:32:26 GMT
Hi Michael,

Any of the tasktracker configs affect the local tasktracker daemon and not
other servers in your cluster.  Moreover, they can't be overridden by a job
configuration.  Sounds like you're in need of a job scheduler; I personally
prefer use the Fair Scheduler but I'm sure the Capacity Scheduler would suit
your needs as well.

cheers,
-James

On Wed, Oct 20, 2010 at 3:41 PM, Michael Moores <mmoores@real.com> wrote:

> I have been playing with mapreduce.tasktracker.map.tasks.maximum to reduce
> the load
> on my Cassandra cluster (using the Cassandra ColumnFamilyInputFormat).  I'd
> like to find ways of throttling the map operations
> in the case I may be affecting OLTP activity on the cluster.
>
> What parameters can I use to limit the number of map tasks running
> concurrently across the whole cluster?
>  mapreduce.tasktracker.map.tasks.maximum
> limits the number of concurrent maps per task tracker.  But can i do this
> at the job level?
>
> Should I look at the "fair" scheduler?
>
> regards,Michael

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message