hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Baldeschwieler <eri...@yahoo-inc.com>
Subject Re: MapReduce: specify a *DFS* path for mapred.jar property
Date Thu, 31 Aug 2006 18:18:58 GMT
Interesting thread.

This relates to HADOOP-288.

Also the thread I started last week on using URLs in general for  
input arguments.  Seems like we should just take a URL for the jar,  
which could be file: or hdfs:

Thoughts?

On Aug 31, 2006, at 10:54 AM, Doug Cutting wrote:

> Frédéric Bertin wrote:
>>> This should run clientside, since it depends on the username,  
>>> which is different on the server.
>> then, what about passing the username as a parameter to the  
>> JobSubmissionProtocol.submitJob(...) ? This avoids loading the  
>> whole JobConf clientside just to set the username.
>
> That sounds like a reasonable change to me.
>
>>>> Why not moving it in the JobSubmissionProtocol (JobTracker's  
>>>> submitJob method) ?
>>>
>>> These could probably run on the server.  They're currently run on  
>>> the client in an attempt to return errors as quickly as possible  
>>> when jobs are misconfigured.
>> Is it really quicker to make all those checkings remotely than  
>> remotely asking the JobTracker to make them locally? (just a  
>> question, I really have no idea of the answer)
>
> We'd need to be careful that this is not a synchronized method on  
> the server, so it doesn't interfere with other server activities.   
> Also, checking the input and output has to be much faster than the  
> RPC timeout, which it should be, since this just checks for the  
> existence of directories, not of individual files.
>
> Doug


Mime
View raw message