hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Frédéric Bertin <frederic.ber...@anyware-tech.com>
Subject Re: MapReduce: specify a *DFS* path for mapred.jar property
Date Thu, 31 Aug 2006 15:29:57 GMT
Doug Cutting wrote:
> Better yet would be to add a new method to JobClient, something like:
>
> public RunningJob submitJob(Path path, FileSystem fs);
>
> Then, if the FileSystem is the same as the JobTracker's, this would 
> directly call JobSubmissionProtocol.submitJob(), otherwise it would do 
> what the current code does.
yes indeed, it looks fine and simple.

I intended to submit a patch with this change, but I have a concern 
about the following code (currently included in submitJob on the 
JobClient) :

>             *// Set the user's name and working directory*
>             String user = System.getProperty("user.name");
>             job.setUser(user != null ? user : "Dr Who");
>             if (job.getWorkingDirectory() == null) {
>               job.setWorkingDirectory(fs.getWorkingDirectory());          
>             }
>
>             FileSystem userFileSys = FileSystem.get(job);
>             Path[] inputDirs = job.getInputPaths();
>             boolean[] validDirs =
>              
>     job.getInputFormat().areValidInputDirectories(userFileSys, inputDirs);
>             for(int i=0; i < validDirs.length; ++i) {
>               if (!validDirs[i]) {
>                 String msg = "Input directory " + inputDirs[i] +
>                              " in " + userFileSys.getName() + " is
>     invalid.";
>                 LOG.error(msg);
>                 throw new IOException(msg);
>               }
>             }
>
>             *// Check the output specification*
>             job.getOutputFormat().checkOutputSpecs(fs, job);

Why not moving it in the JobSubmissionProtocol (JobTracker's submitJob 
method) ?
Indeed, executing this code on the JobClient machine requires to have a 
local copy of the job's config file, and also of the job's Jar file if 
the job uses custom InputFormat or OutputFormat classes which are not 
included in the Hadoop's Jar.

WDYT?

Thanks,

Fred



Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message