hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From kartheek muthyala <kartheek0...@gmail.com>
Subject Re: Job Scheduler, Task Scheduler and Fair Scheduler
Date Sat, 17 Sep 2011 08:24:11 GMT
Hey Arun,
Thanks for the information. And sorry for my previous mail regarding
updates!! I just wanted to emphasize the importance of the query. I couldn't
get enough time to go through the code that's why i approached you guys, as
you are expertise in this area.
Thanks & Regards,
Kartheek.

On Sat, Sep 17, 2011 at 12:09 PM, Arun C Murthy <acm@hortonworks.com> wrote:

>
> On Sep 16, 2011, at 11:26 PM, kartheek muthyala wrote:
>
> > Any updates!!
>
> A bit of patience will help. It also helps to do some homework and ask
> specific questions.
>
> I don't know if you have looked at any of the code, but there are 3
> schedulers:
> JobQueueTaskScheduler (aka default scheduler or fifo scheduler)
> Capacity Scheduler (CS)
> Fair Scheduler (FS).
>
> TaskScheduler is just an interface for all schedulers (default, CS, FS).
>
> Then there is JobInProgress which handles scheduling for map tasks of an
> individual job based on data locality (JobInProgress.obtainNew*MapTask).
>
> Other than that each of the schedulers (default, CS, FS) use different
> criteria for picking a certain job to offer a 'slot' on a given TT when it's
> available.
>
> All this has changed radically and completely with MRv2 which is now in
> branch-0.23 and trunk to allow MR and non-MR apps on same Hadoop cluster:
> http://wiki.apache.org/hadoop/NextGenMapReduce
>
> Arun
>
> >
> > ---------- Forwarded message ----------
> > From: kartheek muthyala <kartheek0274@gmail.com>
> > Date: Fri, Sep 16, 2011 at 8:38 PM
> > Subject: Job Scheduler, Task Scheduler and Fair Scheduler
> > To: common-user@hadoop.apache.org
> >
> >
> > Hi all,
> > Can any one explain me the responsibilities of each scheduler?. I am
> > interested in the flow of commands that goes between these scheduler. And
> if
> > any one have any info regarding how the job scheduler schedules a job
> based
> > on the data locality?. As of I know, there is some heartbeat mechanism
> that
> > goes from task scheduler to job scheduler and in response job scheduler
> does
> > something here to find out the node where the data is more closely
> located
> > and schedules the task in that node. Is there an elaborate way of
> > explanation around this area?. Any help will be greatly appreciated.
> > Thanks and Regards,
> > Kartheek.
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message