hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chen He <airb...@gmail.com>
Subject Re: Yarn / mapreduce scheduling
Date Tue, 26 Jan 2016 22:59:47 GMT
Hi Brad

IMHO, here is how hadoop scheduler works

For Hadoop 1.x

In one heartbeat interval, Fifo Scheduler keep strict FIFO order and assign
a node as many as possible local task and only one remote task. For Fair
scheduler, if you turn on delay algorithm, it also assign local task as
many as possible but none remote task if a job does not wait longer than a
threshold. If a job wait longer than a threshold, fair scheduler will
assign remote task to this job.

For Hadoop 2.x

In one heartbeat interval, Fifoscheduler does not keep the strict FIFO
order, it assign local containers to a AM on a given node, if this AM
blacklist a node, Fifoscheduler will skip this AM and check the next AM in
the job queue. For data locality, FifoScheduler is sharing the same policy
with Capacityscheduler since they are both using
FiCaSchedulerApp.allocate() to assign container.

Hope it is useful



On Tue, Jan 26, 2016 at 2:04 PM, Sultan <sul1418@gmail.com> wrote:

> Brad Childs <bdc@...> writes:
> >
> > Sorry if this is the wrong list, i am looking for deep technical/hadoop
> source help :)
> >
> > How does job scheduling work on yarn framework for map reduce jobs?  I
> see
> the yarn scheduler discussed here:
> >
> https://hadoop.apache.org/docs/r2.2.0/hadoop-yarn/hadoop-yarn-site/YARN.html
>  which leads me
> > to believe tasks are scheduled based on node capacity and not data
> locality.  I've sifted through the fair
> > scheduler and can't find anything about data location or locality.
> >
> > Where does data locality play into the scheduling of map/reduce tasks on
> yarn?  Can someone point me to the
> > hadoop 2.x source where the data block location is used to calculate
> node/container/task assignment (if
> > thats still happening).
> >
> > -bc
> >
> >
> Hi Brad,
> Were you able to find an answer for you question?
> Sultan

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message