hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Venu Gopala Rao <venugopalarao.ko...@huawei.com>
Subject RE: Quincy Fair Scheduler Vs Hadoop Fair Scheduler
Date Mon, 20 Jun 2011 13:21:40 GMT
Thanks Matei.

-----Original Message-----
From: Matei Zaharia [mailto:matei@eecs.berkeley.edu] 
Sent: Saturday, June 18, 2011 4:18 AM
To: mapreduce-dev@hadoop.apache.org
Subject: Re: Quincy Fair Scheduler Vs Hadoop Fair Scheduler

Hi Venu,

The paper compares against a naive implementation of fair scheduling, not
against the Hadoop Fair Scheduler. You can find out about the data locality
algorithms used in the Fair Scheduler in
http://www.cs.berkeley.edu/~matei/papers/2010/eurosys_delay_scheduling.pdf.
These are available in 0.21+, and they have also been in various
distributions of Hadoop (e.g. Cloudera's and Facebook's) for a while. I
don't think they got backported into 0.20.203 though and I'm not sure what
scheduling algorithm the next-gen team is doing for locality.

Matei

On Jun 17, 2011, at 6:46 AM, Venu Gopala Rao wrote:

> Hi All,
> 
> 
> 
>   I have come across a Fair Scheduler published by Microsoft known as
> Quincy Fair SCheduler. In this they compare Hadoop Fair Scheduler with
> Quincy and say the Hadoop Scheduler has the following problems
> 
> 
> 
> 1)  Sticky Slots - One drawback of simple fair scheduling is that it is
> damaging to locality. Consider the steady state in which each job is
> occupying exactly its allocated quota of computers. Whenever a task from
job
> j completes on computer m, j becomes unblocked but all of the other jobs
in
> the system remain blocked. Consequently m will be assigned to one of j's
> tasks again: this is referred to as the "sticky slot" problem.
> 
> 
> 
> 2) Fair Scheduler may not be able to utilize the Data locality to maximum
> possible extent.
> 
> 
> 
> Does these problems get solved in the 0.21 or Next Gen Map Reduce?
> 
> 
> 
> Regards
> 
> Venu
> 
> 
> 
> 
> 
> 
> 


Mime
View raw message