hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Segel, Mike" <mse...@navteq.com>
Subject RE: Task scheduler
Date Mon, 17 May 2010 12:19:11 GMT
+1
I agree with Steve that sometimes you need to redirect where you want the work to occur.

Over time, your cloud will not have homogenous data nodes. You may end up with a cluster of
nodes that have a Fermi card (NVIDA CUDA enabled cards) where you want to do some serious
number crunching. [ I don't believe if CUDA supports Java, but you get the idea...]

So in theory, you will want to dictate where the work will be performed.

-Mike

PS. Sorry if this is a bad example. I was talking with my Uncle over the weekend and he's
in to some serious number crunching.

-----Original Message-----
From: Steve Loughran [mailto:stevel@apache.org] 
Sent: Monday, May 17, 2010 6:47 AM
To: common-dev@hadoop.apache.org
Subject: Re: Task scheduler

Saurabh Agarwal wrote:
> Hemanth,
> 
> 
> Thanks!!
> Saurabh Agarwal
> 
> 
> On Fri, May 14, 2010 at 9:49 AM, Hemanth Yamijala <yhemanth@gmail.com>wrote:
> 
>> Saurabh,
>>
>>>  let me re frame my question I wanted to knowhow job tracker decides the
>>> assignment of input splits to task tracker based on task tracker's data
>>> locality. Where is this policy defined? Is it pluggable?
>> Sorry, I misunderstood your question then. This code is in
>> o.a.h.mapred.JobInProgress. It is likely spread across many methods in
>> the class. But a good starting point could be from methods like
>> obtainNewMapTask or obtainNewReduceTask.
>>
>> At the moment, this policy is not pluggable. But I know there have
>> been discussions (possibly even a JIRA, though I can't locate any now)
>> asking for this capability.
>>

+1 to having some plugin interface in 0.22+ to give you control.

My fomer colleague russ perry did some rendering with Hadoop where he 
wanted the work done not where the input data was, but where the output 
data was needed; there was no way to do this
http://www.hpl.hp.com/techreports/2009/HPL-2009-345.pdf



The information contained in this communication may be CONFIDENTIAL and is intended only for
the use of the recipient(s) named above.  If you are not the intended recipient, you are hereby
notified that any dissemination, distribution, or copying of this communication, or any of
its contents, is strictly prohibited.  If you have received this communication in error, please
notify the sender and delete/destroy the original message and any copy of it from your computer
or paper files.

Mime
View raw message