hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nan Zhu <zhunans...@gmail.com>
Subject Re: Question about Hadoop Default FCFS Job Scheduler
Date Mon, 17 Jan 2011 14:28:20 GMT
Hi, Chen

How is it going recently?

Actually I think you misundertand the code in assignTasks() in
JobQueueTaskScheduler.java, see the following structure of the interesting

//I'm sorry, I hacked the code so much, the name of the variables may be
different from the original version

for (i = 0; i < MapperCapacity; ++i){
   for (JobInProgress job:jobQueue){
       //try to shedule a node-local or rack-local map tasks
       //here is the interesting place
       t = job.obtainNewLocalMapTask(...);
       if (t != null){
          break;//the break statement here will make the control flow back
to "for (job:jobQueue)" which means that it will restart map tasks selection
procedure from the first job, so , it is actually schedule all of the first
job's local mappers first until the map slots are full

BTW, we can only schedule a reduce task in a single heartbeat

On Sat, Jan 15, 2011 at 1:45 PM, He Chen <airbots@gmail.com> wrote:

> Hey all
> Why does the FCFS scheduler only let a node chooses one task at a time in
> one job? In order to increase the data locality,
> it is reasonable to let a node to choose all its local tasks (if it can)
> from a job at a time.
> Any reply will be appreciated.
> Thanks
> Chen

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message