Mailing-List: contact core-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: core-user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of annndy.lee@gmail.com
 designates 72.14.220.154 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:to:subject:in-reply-to:mime-version:content-type:references;
        b=qGSoS2moB0/A4jYV9LLpblE1a2/FkDfWnv0xRlGvZoKLNrTgDoAp232GBIue9NpZH4Q+dLfvPPRkAXeQ5+aeGjRunghKq/+gKQ0vcL4egnQUNVbKaS/6Q9NWSZFFH7NuSHTw6X/sNeMvUXsC0dsvubpz2uDtLBYaDUjHtf4wVh8=
Message-ID: <8e8d18110802152117g42df2923lc8cab5faca7cc69b@mail.gmail.com>
Date: Fri, 15 Feb 2008 21:17:59 -0800
From: "Andy Li" <annndy.lee@gmail.com>
To: core-user@hadoop.apache.org
Subject: Re: Questions about the MapReduce libraries and job schedulers inside
 JobTracker and JobClient running on Hadoop
In-Reply-To: <00aa01c87026$b90b68e0$652317ac@ds.corp.yahoo.com>
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----=_Part_5122_7602549.1203139079428"
References: 
 <9FA876DC4532C74BBA9EB08E97F9BA1501A6E7DA@UDCEXMAIL01.us.trendnet.org>
	 <C3DB4A0A.386FB%tdunning@veoh.com>
	 <00aa01c87026$b90b68e0$652317ac@ds.corp.yahoo.com>

------=_Part_5122_7602549.1203139079428
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

Thanks for both inputs.  My question actually focus more on what Vivek has
mentioned.

I would like to work on the JobClient to see how it submits jobs to
different file system and
slaves in the same Hadoop cluster.

Not sure if there is a complete document to explain the scheduler underneath
Hadoop,
if not, I'll wrap up what I know and study from the source code and submit
it to the community
once it is done.  Review and comments are welcome.

For the code, I couldn't find JobInProgress from the API index.  Could
anyone provide me
a pointer to this?  Thanks.

On Fri, Feb 15, 2008 at 3:01 PM, Vivek Ratan <vivekr@yahoo-inc.com> wrote:

> I read Andy's question a little differently. For a given job, the
> JobTracker
> decides which tasks go to which TaskTracker (the TTs ask for a task to run
> and the JT decides which task is the most appropriate). Currently, the JT
> favors a task whose input data is on the same host as the TT (if there are
> more than one such tasks, it picks the one with the largest input size).
> It
> also looks at failed tasks and certain other criteria. This is very basic
> scheduling and there is a lot of scope for improvement. There currently is
> a
> proposal to support rack awareness, so that if the JT can't find a task
> whose input data is on the same host as the TT, it looks for a task whose
> data is on the same rack.
>
> You can clearly get more ambitious with your scheduling algorithm. As you
> mention, you could use other criteria for scheduling a task: available CPU
> or memory, for example. You could assign tasks to hosts that are the most
> 'free', or aim to distribute tasks across racks, or try some other load
> balancing techniques. I believe there are a few discussions on these
> methods
> on Jira, but I don't think there's anything concrete yet.
>
> BTW, the code that decides what task to run is primarily in
> JobInProgress::findNewTask().
>
>
> -----Original Message-----
> From: Ted Dunning [mailto:tdunning@veoh.com]
> Sent: Friday, February 15, 2008 1:54 PM
> To: core-user@hadoop.apache.org
> Subject: Re: Questions about the MapReduce libraries and job schedulers
> inside JobTracker and JobClient running on Hadoop
>
>
> Core-user is the right place for this question.
>
> Your description is mostly correct.  Jobs don't necessarily go to all of
> your boxes in the cluster, but they may.
>
> Non-uniform machine specs are a bit of a problem that is being (has been?)
> addressed by allowing each machine to have a slightly different
> hadoop-site.xml file.  That would allow different settings for storage
> configuration and number of processes to run.
>
> Even without that, you can level the load a bit by simply running more
> jobs
> on the weak machines than you would otherwise prefer.  Most map reduce
> programs are pretty light on memory usage so all that happens is that you
> get less throughput on the weak machines.  Since there are normally more
> map
> tasks than cores, this is no big deal; slow machines get fewer tasks and
> toward the end of the job, their tasks are even replicated on other
> machines
> in case they can be done more quickly.
>
>
> On 2/15/08 1:25 PM, "Andrew_Lee@trendmicro.com" <Andrew_Lee@trendmicro.com
> >
> wrote:
>
> >
> > Hello,
> >
> > My first time posting this in the news group.    My question sounds more
> like
> > a MapReduce question
> > instead of Hadoop HDFS itself.
> >
> > To my understanding, the JobClient will submit all Mapper and Reduce
> > class in a uniform way to the cluster?  Can I assume this is more like
> > a uniform scheduler for all the task?
> >
> > For example, if I have a 100 node cluster, 1 master (namenode), 99
> > slaves (datanodes).
> > When I do
> > "JobClient.runJob(jconf)"
> > the JobClient will uniformly distributes all Mapper and Reduce class
> > to all 99 nodes.
> >
> > In the slaves, they will all have the same hadoop-site.xml and
> > hadoop-default.xml.
> > Here comes the main concern, what if some of the nodes don't have the
> > same hardware spec such as memory or CPU speed?  E.g. different batch
> > purchase and repairment overtime that causes this.
> >
> > Is there any way that the JobClient can be aware of this and submit
> > different number of tasks to different slaves during start-up?
> > For example, for some slaves, it has 16 cores CPU instead of 8 cores.
> > The problem I see here is that for the 16 cores, only 8 cores are
> > used.
> >
> > P.S. I'm looking into the JobClient source code and
> > JobProfile/JobTracker to see if this can be done.
> > But not sure if I am on the right track.
> >
> > If this topic is more likely to be in the core-dev@hadoop.apache.org,
> > please let me know.  I'll send another one to that news group.
> >
> > Regards,
> > -Andy
> >
> > TREND MICRO EMAIL NOTICE
> > The information contained in this email and any attachments is
> > confidential and may be subject to copyright or other intellectual
> > property protection. If you are not the intended recipient, you are
> > not authorized to use or disclose this information, and we request
> > that you notify us by reply mail or telephone and delete the original
> message from your mail system.
>
>
>

------=_Part_5122_7602549.1203139079428--