hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From radheshyam nanduri <radheshyam.nand...@gmail.com>
Subject Re: Launching Tasks From JobTracker
Date Fri, 17 Sep 2010 08:30:58 GMT
hi,

Thanks for the reply.

The problem i want to solve is, I am writing a scheduler plugin in which, i
want to divide a job into tasks and assign to TaskTrackers(Obvious).

Then, i want to assign the tasks on to which ever TaskTracker i want and
assign it whenever i want.

I am using
org.apache.hadoop.mapred.TaskScheduler.assignTasks(TaskTrackerStatus
taskTrackerStatus) which would return list of Tasks, on every Heart Beat.

In the above method, i am calling
org.apache.hadoop.mapred.JobInProgress.obtainNewMapTask method to get Task
for a job. But everytime it is called, i would only get a null as return.

Initially i thought that JobStatus.status is not set to RUNNING.
But the problem is not that.

Is there any other way that i could assign tasks on to a TaskTracker apart
from above method.


hoping a solution from you.

Thank you

Radheshyam Nanduri

On Fri, Sep 10, 2010 at 6:46 PM, Hemanth Yamijala <yhemanth@gmail.com>wrote:

> [Moving to mapreduce-dev, copying common-dev]
>
> Hi,
>
> On Thu, Sep 9, 2010 at 11:30 AM, radheshyam nanduri
> <radheshyam.nanduri@gmail.com> wrote:
> > Hi,
> >
> > I am working on writing a scheduler plugin for Hadoop.
>
> Currently, the model supported to plug-in schedulers to Hadoop is to
> extend the TaskScheduler class in o.a.h.mapred package. Primarily what
> a 'plug-in' scheduler can do is that given a set of jobs and a
> tasktracker, it can assign one or more suitable tasks to the
> tasktracker. The scheduler will have flexibility in choosing the job
> and the tasks it wants to schedule. You can take a look at some of the
> existing schedulers like CapacityTaskScheduler or FairScheduler to see
> what they do and how.
>
> > I have divided the job received into number of tasks.
>
> This is already done in the framework when a job is submitted. Are you
> overriding this ? Can you explain what you are doing in some more
> detail ?
>
> > My task now is to assign a task on to a particular TaskTracker.
> > I want to start the Task right away with a method which accepts the Task
> and
> > TaskTracker as arguments.
>
> I am not sure I am following this. A task needs to be sent via
> Hadoop's RPC mechanisms to a tasktracker where it should be executed.
> So, conceptually, it is the tasktracker that has an RPC method which
> accepts tasks to launch. The task is launched typically straight-away,
> but in case of certain scheduling choices, it could have to wait for a
> short while to get a free slot to execute.
>
> > Could you guide me on doing this.
>
> It may be better if you can describe what you want the plug-in
> scheduler to achieve.
>
> Thanks
> Hemanth
>
> >
> >
> > Thanks in advance.
> >
> >
> > --
> > Radheshyam Nanduri
> >
>



-- 
Radheshyam Nanduri

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message