hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Doug Cutting <cutt...@apache.org>
Subject Re: MapReduce: specify a *DFS* path for mapred.jar property
Date Wed, 30 Aug 2006 16:07:48 GMT
Frédéric Bertin wrote:
> sorry if I missed something obvious, but after looking at the JobClient 
> and JobTracker code, it seems to me that there is no way for the JobClient
> to submit a job to the JobTracker using a Jar file which is already 
> located on the JobTracker's DFS. The "mapred.jar" property seems to be 
> interpreted as
> a local path (from the JobClient's point of view), which necessarily 
> leads to a file transfer from the JobClient to the tracker's DFS.
> Am I right?

Yes, although this should be easy to repair, since the restriction is 
confined to JobClient code.  When the JobClient submits the job to the 
job tracker, it sends only the path of a DFS-resident config file which 
names a DFS-resident jar file.

> Is it possible to achieve this with the current Hadoop code?

Yes, but it would be a hack.  One could write some code that directly 
invokes JobSubmissionProtocol in the mapred package (since 
JobSubmissionProtocol is package-private).

Better yet would be to add a new method to JobClient, something like:

public RunningJob submitJob(Path path, FileSystem fs);

Then, if the FileSystem is the same as the JobTracker's, this would 
directly call JobSubmissionProtocol.submitJob(), otherwise it would do 
what the current code does.

Doug

Mime
View raw message