hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michel Segel <michael_se...@hotmail.com>
Subject Re: Reducer granularity and starvation
Date Thu, 19 May 2011 11:51:23 GMT
Fair scheduler won't help unless you set it to allow preemptive executions which may not be
a good thing...

Fair scheduler will wait until the current task completes before assigning a new task to the
open slot.  So if you have a long running job... You're SOL.

A combiner will definitely help but you will still have the issue of long running jobs. You
could put you job in a queue that limits the number of slots... But then you will definitely
increase the time to run your job.

If you could suspend a task... But that's anon-trivial solution...

Sent from a remote device. Please excuse any typos...

Mike Segel

On May 18, 2011, at 5:04 PM, "W.P. McNeill" <billmcn@gmail.com> wrote:

> I'm using fair scheduler and JVM reuse.  It is just plain a big job.
> I'm not using a combiner right now, but that's something to look at.
> What about bumping the mapred.reduce.tasks up to some huge number?  I think
> that shouldn't make a difference, but I'm hearing conflicting information on
> this.

View raw message