hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matei Zaharia <ma...@cloudera.com>
Subject Re: Running only node-local jobs
Date Sat, 19 Dec 2009 03:32:45 GMT
Hi Ives,

It's actually pretty easy to do this with the Fair Scheduler in trunk. You
can just set the mapred.fairscheduler.locality.delay config property to a
large number, say 1000000000000. This parameter controls how long the
scheduler waits before it is allowed to launch non-local tasks from a given
job. Normally it is set to 1-5 seconds, which is enough to get high locality
for most jobs without risking starvation if a node is full of long tasks.
But setting it to a high number like this forces only local scheduling.

This feature uses some APIs from 0.21 which aren't present in 0.20. However,
if you want to use this behavior in 0.20, you might be able to use an older
version of my patches from
https://issues.apache.org/jira/browse/MAPREDUCE-548. (Basically the main
thing needed is an extra parameter to JobInProgress.obtainNewMapTask.) Also,
according to https://issues.apache.org/jira/browse/MAPREDUCE-1129, Cloudera
might have included my scheduler patch in version 3 of their distribution.


On Fri, Dec 18, 2009 at 7:24 AM, Ives Aerts
> wrote:

> On Thu, Dec 17, 2009 at 8:24 PM, Owen O'Malley <omalley@apache.org> wrote:
> > On Dec 17, 2009, at 2:06 AM, Ives Aerts wrote:
> ]>> Quick question: is it possible to configure a job such that its map
> >> tasks will only run node-local?
> >
> > No, that is currently not possible without writing your own scheduler.
> Too bad. Now I wouldn't mind writing my own scheduler (or subclassing
> an existing one) and plugging it in but I had a quick look and it
> seems that an important part of the task scheduling logic is actually
> contained in the JobInProgress class which is instantiated by the
> JobTracker. So even if I replace the scheduler I won't be able to
> override that part of the functionality. Or am I missing something?
> --
> Cheers,
> -Ives

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message