hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dan Adkins <dadk...@quantcast.com>
Subject non-existing directories not ignored
Date Wed, 08 Sep 2010 19:20:40 GMT
According to the description of this property, directories that do not exist are ignored.

<property>
  <name>mapreduce.cluster.local.dir</name>
  <value>${hadoop.tmp.dir}/mapred/local</value>
  <description>The local directory where MapReduce stores intermediate             
                      
  data files.  May be a comma-separated list of                                          
                
  directories on different devices in order to spread disk i/o.                          
                
  Directories that do not exist are ignored.                                             
                
  </description>
</property>

However, on my system, that's clearly not the case:

10/09/08 19:11:59 WARN mapred.TaskTracker: Task Tracker local can not create directory: /mnt/data8/mr20
10/09/08 19:11:59 ERROR mapred.TaskTracker: Can not start task tracker because java.io.IOException:
Cannot create toBeDeleted in /mnt/data8/mr20
	at org.apache.hadoop.mapreduce.util.MRAsyncDiskService.<init>(MRAsyncDiskService.java:88)
	at org.apache.hadoop.mapreduce.util.MRAsyncDiskService.<init>(MRAsyncDiskService.java:118)
	at org.apache.hadoop.mapred.TaskTracker.initialize(TaskTracker.java:618)
	at org.apache.hadoop.mapred.TaskTracker.<init>(TaskTracker.java:1299)
	at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:3461)

Drives fail.  The task tracker shouldn't fail to start because one of them isn't present.
 Is this a bug in the code, bug in the description, or is there some other property that I
need to set to make this happen?

-Dan


Mime
View raw message