hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yoram Arnon (JIRA)" <j...@apache.org>
Subject [jira] Created: (HADOOP-242) job fails because of "No valid local directories in property: " exception
Date Tue, 23 May 2006 19:15:29 GMT
job fails because of "No valid local directories in property: " exception
-------------------------------------------------------------------------

         Key: HADOOP-242
         URL: http://issues.apache.org/jira/browse/HADOOP-242
     Project: Hadoop
        Type: Bug

  Components: mapred  
    Reporter: Yoram Arnon
 Assigned to: Sameer Paranjpye 


when running a fairly large job, of 70+K map tasks, I get many exceptions as shown below,
and eventually the job failes when a task fails four times.
The exception doesn't really tell us enough information to debug this properly, so the first
thing to do would be to add more information (path) to the exception.
The path indicated in the config file exists, is writable and valid, though 'path' may be
anything.

the exception:
java.io.IOException: No valid local directories in property: mapred.local.dir at org.apache.hadoop.conf.Configuration.getLocalPath(Configuration.java:293)
at org.apache.hadoop.mapred.JobConf.getLocalPath(JobConf.java:153) at org.apache.hadoop.mapred.TaskTracker$TaskInProgress.localizeTask(TaskTracker.java:523)
at org.apache.hadoop.mapred.TaskTracker$TaskInProgress.launchTask(TaskTracker.java:572) at
org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:389) at org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:303)
at org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:418) at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:920)

the code:
  public Path getLocalPath(String dirsProp, String path)
    throws IOException {
    String[] dirs = getStrings(dirsProp);
    int hashCode = path.hashCode();
    FileSystem fs = FileSystem.getNamed("local", this);
    for (int i = 0; i < dirs.length; i++) {  // try each local dir
      int index = (hashCode+i & Integer.MAX_VALUE) % dirs.length;
      Path file = new Path(dirs[index], path);
      Path dir = file.getParent();
      if (fs.exists(dir) || fs.mkdirs(dir)) {
        return file;
      }
    }
    throw new IOException("No valid local directories in property: "+dirsProp);
  }


-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira


Mime
View raw message