hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From patek tek <patek...@gmail.com>
Subject Re: Scheduler recommendation
Date Thu, 12 Aug 2010 12:55:53 GMT
I have been trying to post to the mailing list without success since
yesterday. This is yet another attempt.


On Thu, Aug 12, 2010 at 1:30 AM, Hemanth Yamijala <yhemanth@gmail.com>wrote:

> Hi,
>
> On Thu, Aug 12, 2010 at 10:31 AM, Hemanth Yamijala <yhemanth@gmail.com>
> wrote:
> > Hi,
> >
> > On Thu, Aug 12, 2010 at 3:35 AM, Bobby Dennett
> > <bdennett+software@gmail.com <bdennett%2Bsoftware@gmail.com>> wrote:
> >> From what I've read/seen, it appears that, if not the "default"
> >> scheduler, most installations are using Hadoop's Fair Scheduler. Based
> >> on features and our requirements, we're leaning towards using the
> >> Capacity Scheduler; however, there is some concern that it may not be
> >> as "stable" as there doesn't appear to be as much talk about it,
> >> compared to the Fair Scheduler.
> >>
> >> Has anyone hit any nasty issues with regards to the Capacity Scheduler
> >> and, in general, are there any "gotchas" to look out for with either
> >> scheduler?
> >>
> >> We're ramping up the number of users on our Hadoop clusters,
> >> particularly in regards to Hive. Our goal is to ensure that production
> >> processes continue to run with a majority of the cluster during peak
> >> usage times, while personal users share the remaining capacity. The
> >> Capacity Scheduler's support of queues and for memory-intensive jobs
> >> is appealing but we are curious about drawbacks and/or potential
> >> issues.
> >
> > FWIW, Yahoo! is running capacity scheduler for a reasonably long time
> > now. However, there have been many patches on top of the base Hadoop
> > 0.20.2 version to capacity scheduler that make it 'stable' and work at
> > large scale effectively. Looking at the change log of the yahoo hadoop
> > distribution could possibly give an idea of which patches are useful
> > to pick up and apply to an older version. The good news is that most
> > of these patches have 0.20 versions that are available on JIRA and
> > would apply reasonably cleanly.
> >
>
> Allen cautions the part about patches applying cleanly to 0.20 might
> not be very true. Thanks for that heads-up, Allen !
>
> Thanks
> Hemanth
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message