hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Scott Whitecross <sc...@dataxu.com>
Subject java.io.IOException: Could not get block locations. Aborting...
Date Mon, 09 Feb 2009 21:01:10 GMT
Hi all -

I've been running into this error the past few days:
java.io.IOException: Could not get block locations. Aborting...
	at org.apache.hadoop.dfs.DFSClient 
	at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access 
	at org.apache.hadoop.dfs.DFSClient$DFSOutputStream 

It seems to be related to trying to write to many files to HDFS.  I  
have a class extending  
org.apache.hadoop.mapred.lib.MultipleOutputFormat and if I output to a  
few file names, everything works.  However, if I output to thousands  
of small files, the above error occurs.  I'm having trouble isolating  
the problem, as the problem doesn't occur in the debugger unfortunately.

Is this a memory issue, or is there an upper limit to the number of  
files HDFS can hold?  Any settings to adjust?

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message