hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Frédéric Bertin <frederic.ber...@anyware-tech.com>
Subject Re: MapReduce: specify a *DFS* path for mapred.jar property
Date Thu, 31 Aug 2006 16:48:42 GMT
Doug Cutting wrote:
> Frédéric Bertin wrote:
>>>             *// Set the user's name and working directory*
>>>             String user = System.getProperty("user.name");
>>>             job.setUser(user != null ? user : "Dr Who");
>>>             if (job.getWorkingDirectory() == null) {
>>>               
>>> job.setWorkingDirectory(fs.getWorkingDirectory());                      
>>> }
>
> This should run clientside, since it depends on the username, which is 
> different on the server.
then, what about passing the username as a parameter to the 
JobSubmissionProtocol.submitJob(...) ? This avoids loading the whole 
JobConf clientside just to set the username.
>
>>>             FileSystem userFileSys = FileSystem.get(job);
>>>             Path[] inputDirs = job.getInputPaths();
>>>             boolean[] validDirs =
>>>                  
>>> job.getInputFormat().areValidInputDirectories(userFileSys, inputDirs);
>>>             for(int i=0; i < validDirs.length; ++i) {
>>>               if (!validDirs[i]) {
>>>                 String msg = "Input directory " + inputDirs[i] +
>>>                              " in " + userFileSys.getName() + " is
>>>     invalid.";
>>>                 LOG.error(msg);
>>>                 throw new IOException(msg);
>>>               }
>>>             }
>>>
>>>             *// Check the output specification*
>>>             job.getOutputFormat().checkOutputSpecs(fs, job);
>>
>> Why not moving it in the JobSubmissionProtocol (JobTracker's 
>> submitJob method) ?
>
> These could probably run on the server.  They're currently run on the 
> client in an attempt to return errors as quickly as possible when jobs 
> are misconfigured.
Is it really quicker to make all those checkings remotely than remotely 
asking the JobTracker to make them locally? (just a question, I really 
have no idea of the answer)

Fred

Mime
View raw message