hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Doug Cutting <cutt...@apache.org>
Subject Re: MapReduce: specify a *DFS* path for mapred.jar property
Date Thu, 31 Aug 2006 17:54:50 GMT
Frédéric Bertin wrote:
>> This should run clientside, since it depends on the username, which is 
>> different on the server.
> then, what about passing the username as a parameter to the 
> JobSubmissionProtocol.submitJob(...) ? This avoids loading the whole 
> JobConf clientside just to set the username.

That sounds like a reasonable change to me.

>>> Why not moving it in the JobSubmissionProtocol (JobTracker's 
>>> submitJob method) ?
>>
>> These could probably run on the server.  They're currently run on the 
>> client in an attempt to return errors as quickly as possible when jobs 
>> are misconfigured.
> Is it really quicker to make all those checkings remotely than remotely 
> asking the JobTracker to make them locally? (just a question, I really 
> have no idea of the answer)

We'd need to be careful that this is not a synchronized method on the 
server, so it doesn't interfere with other server activities.  Also, 
checking the input and output has to be much faster than the RPC 
timeout, which it should be, since this just checks for the existence of 
directories, not of individual files.

Doug

Mime
View raw message