hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ahmad Humayun" <ahmad.hu...@gmail.com>
Subject Re: About the support of MapReduce task scheduling schemes
Date Wed, 13 Feb 2008 17:59:42 GMT
As far as I understand, since this is a distributed computation, you wouldnt
find scheduling at one place. Everything is running as concurrent threads on
separate machines. But you might want to look at the JobTracker (the Master
process) ... you might want to look at its sub-class JobInitThread and see
what things are being run from there on, especially
JobInProgress::initTasks(). Also look at the TaskTracker which manages the
running of all Map and Reduce functions at a specific machine.


hope this helps,



On Feb 13, 2008 10:32 PM, Zhu Huijun <nautilushj@gmail.com> wrote:

> Thanks. Do you have any idea about where is the scheduling module in the
> source files?
>
> 2008/2/13, Ahmad Humayun <ahmad.humyn@gmail.com>:
> >
> > Its not a separate module, as far as I know, its just part of the whole
> > Hadoop implementation. If you would like to use it somewhere else, it
> > would
> > take some serious surgery :)
> >
> > I'm sure you didn't want to hear this, but this is what I know. Although
> > I'll recommend you to ask others too :)
> >
> > regards,
> >
> > On Feb 13, 2008 10:18 PM, Zhu Huijun <nautilushj@gmail.com> wrote:
> >
> > > Thank you, Ahmad Humayun. I am not asking for the idea of MapReduce.
> > What
> > > I
> > > am asking is the scheduling scheme in Hadoop. I am wondering whether
> the
> > > scheduling module is a part of any library implemented in Hadoop, or
> the
> > > scheduling of Hadoop is based on the infrastructure under which Hadoop
> > is
> > > installed.
> > >
> > > 2008/2/13, Ahmad Humayun <ahmad.humyn@gmail.com>:
> > > >
> > > > The ideas were originally published as MapReduce by Jeffrey
> > > > Dean<http://labs.google.com/people/jeff>and Sanjay
> > > > Ghemawat <http://labs.google.com/people/sanjay> at Google. Have
a
> look
> > > at:
> > > > http://labs.google.com/papers/mapreduce.html
> > > >
> > > >
> > > > regards,
> > > >
> > > > On Feb 13, 2008 9:21 PM, Zhu Huijun <nautilushj@gmail.com> wrote:
> > > >
> > > > > Dear all,
> > > > >
> > > > > Recently we are invesgating dynamic scheduling of parallel
> > > IO-intensive
> > > > > applications in large-scale cluster. We are interested in Hadoop,
> > > > > especially
> > > > > its task scheduling schemes. Is the scheduling module a part of
> any
> > > > > library
> > > > > of Hadoop, or is it a standalone library? Is there any
> publications
> > > > > specific
> > > > > on scheduling in Hadoop? Could you please share some details about
> > > > > scheduling or suggest some literature of Hadoop?
> > > > >
> > > > > Thanks!
> > > > >
> > > > > Best wishes,
> > > > >
> > > > > Huijun Zhu
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Ahmad Humayun
> > > > Research Assistant
> > > > Computer Science Dpt., LUMS
> > > > +92 321 4457315
> > > >
> > >
> >
> >
> >
> > --
> > Ahmad Humayun
> > Research Assistant
> > Computer Science Dpt., LUMS
> > +92 321 4457315
> >
>



-- 
Ahmad Humayun
Research Assistant
Computer Science Dpt., LUMS
+92 321 4457315

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message