hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <qwertyman...@gmail.com>
Subject Re: mapper and reducer scheduling
Date Fri, 29 Oct 2010 14:11:02 GMT

On Fri, Oct 29, 2010 at 12:45 PM, Jeff Zhang <zjffdu@gmail.com> wrote:
> TaskTracker will tell JobTracker how many free slots it has through
> heartbeat. And JobTracker will choose the best tasktracker with the
> consideration of data locality.

Yes. To add some more, a scheduler is responsible to do assignments of
tasks (based on various stats, including data locality) to proper
tasktrackers. Scheduler.assignTasks(TaskTracker) is used to assign a
TaskTracker its tasks, and the scheduler type is configurable (Some
examples are Eager/FIFO scheduler, Capacity scheduler, etc.).

This scheduling is done when a heart beat response is to be sent back
to a TaskTracker that called JobTracker.heartbeat(...).

> On Thu, Oct 28, 2010 at 2:52 PM, Zhenhua Guo <jenvor@gmail.com> wrote:
>> Hi, all
>>  I wonder how Hadoop schedules mappers and reducers (e.g. consider
>> load balancing, affinity to data?). For example, how to decide on
>> which nodes mappers and reducers are to be executed and when.
>>  Thanks!
>> Gerald
> --
> Best Regards
> Jeff Zhang

Harsh J

View raw message