hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ion Badita <ion.bad...@mcr.ro>
Subject Hadoop: No space left on device
Date Tue, 24 Oct 2006 05:43:48 GMT

I did some M/R test on a 8 computers cluster. The mapper just generate 
some random data (8 maps) and one reduce merges it.
The data generated by the maps (intentionally) is larger than the disk 
space available on the reduce computer.
When copying the data from map 2 the reduce computer continuously throw:

org.apache.hadoop.fs.FSError: java.io.IOException: No space left on device
	at org.apache.hadoop.fs.LocalFileSystem$LocalFSFileOutputStream.write(LocalFileSystem.java:161)
	at org.apache.hadoop.fs.FSDataOutputStream$Summer.write(FSDataOutputStream.java:83)
	at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:112)
	at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
	at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
	at java.io.DataOutputStream.flush(DataOutputStream.java:106)
	at java.io.FilterOutputStream.close(FilterOutputStream.java:140)
	at org.apache.hadoop.mapred.MapOutputLocation.getFile(MapOutputLocation.java:127)
	at org.apache.hadoop.mapred.ReduceTaskRunner$MapOutputCopier.copyOutput(ReduceTaskRunner.java:238)
	at org.apache.hadoop.mapred.ReduceTaskRunner$MapOutputCopier.run(ReduceTaskRunner.java:208)
Caused by: java.io.IOException: No space left on device
	at java.io.FileOutputStream.writeBytes(Native Method)
	at java.io.FileOutputStream.write(FileOutputStream.java:260)
	at org.apache

Is there an "safe mechanism" to overcome this kind of problems, maybe 
when there is no space left on the local disk the dfs space may be used, 
with a performance penalty, but the system will continue to work.

Thank You!

View raw message