hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vijay Murthi" <murt...@yahoo-inc.com>
Subject Out of memory after Map tasks
Date Thu, 25 May 2006 17:16:11 GMT
I am trying to understand what happens during the time duration when Map task got finished
and reduce task starts executing. I have 2 machines with 4 process + 4 Gigs on each with NFS
(not dfs) to process 50 Gigs of data. Map taks finish completion successfully. After that
I see the following on the tasktracker log.

"Exception in thread "Server handler 1 on 50040" java.lang.OutOfMemoryError: Java heap space"

Lister below is the configuration parameter. Am I setting JAVA memory heap very low compared
to io.sort.mb or file buffer size? I thought Tasktracker just pushes the job to the child
node, does it because of something like moving data ? If so is there a buffer size I can set
a limit? Also, I noticed on mapred local each under the directotries for reduce files start
growing even after tasktracker has "out of memory error".

Any feedback would be appreciated.













# The maximum amount of heap to use, in MB. Default is 1000.

  • Unnamed multipart/mixed (inline, None, 0 bytes)
View raw message