hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <ha...@cloudera.com>
Subject Re: Technical question on Capacity Scheduler.
Date Sun, 03 Mar 2013 10:41:46 GMT
On Sun, Mar 3, 2013 at 1:41 PM, Jagmohan Chauhan <simplefundumnnit@gmail.com
> wrote:

>  Hi
> I am going through the Capacity Scheduler implementation. There is one
> thing i did not understand clearly.

Are you reading the YARN CapacityScheduler or the older, MRv1 one? I'd
suggest reading the newer one for any implementation or research goals, for
it to be more current and future-applicable.

> 1. Does the o ff-switch task refers to a task in which data has to be
> fetched over the network. It means its not node-local ?

Off-switch would imply off-rack, i.e. not node local, nor rack-local.

> 2. Does off-switch task  includes only the tasks for which map input has to
> be fetched from a node on a different rack across the switch or it also
> includes task where data has to be fetched from another node on same rack
> on same switch?

A task's input split is generally supposed to define all locations of
available inputs. If the CS is unable to schedule to any of those
locations, nor their racks, then it schedules an off-rack (see above) task
which has to pull the input from a different rack.

> --
> Thanks and Regards
> Jagmohan Chauhan
> MSc student,CS
> Univ. of Saskatchewan
> IEEE Graduate Student Member
> http://homepage.usask.ca/~jac735/

Feel free to post any further impl. related questions! :)

Harsh J

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message