hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Doug Cutting <cutt...@apache.org>
Subject Re: Mapper Out of Memory
Date Thu, 06 Dec 2007 22:14:22 GMT
Rui Shi wrote:
> It is hard to believe that you need to enlarge heap size given the input size is only
10MB. In particular, you don't load all input at the same time. As for the program logic,
no much fancy stuff, mostly cut and sorting. So GC should be able to handle...

Out-of-memory exceptions can also be caused by having too many files 
open at once.  What does 'ulimit -n' show?

You presented an excerpt from a jobtracker log, right?  What do the 
tasktracker logs show?

Can you monitor a node while it is running to see whether the jvm's heap 
is growing, or whether the number of open files (lsof -p) is large?

Also, can you please provide more details about your application?  I.e., 
what is your inputformat, map function, etc.


View raw message