hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Christian Kunz (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-3670) JobTracker running out of heap space
Date Thu, 03 Jul 2008 20:07:45 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-3670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12610334#action_12610334
] 

Christian Kunz commented on HADOOP-3670:
----------------------------------------

Because of the suspicion that GC was badly configured, I restarted JobTracker in 32-bit mode
with default configuration, but with the options suggested by Owen:

HADOOP_OPTS="-server  -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+HeapDumpOnOutOfMemoryError"

HADOOP_HEAPSIZE=2500

RAM: 8GB

The cluster has 200 nodes, jobs have typically at most 4000 maps and less than 400 reduces,
but often 2 or 3 jobs run simultaneously.

The JobTracker's memory footprint increased slowly close up to 2.4GB, and then after about
100 jobs a new job initialization failed:

Job initialization failed:
java.lang.OutOfMemoryError: Java heap space
        at org.apache.hadoop.io.BytesWritable.setCapacity(BytesWritable.java:97)
        at org.apache.hadoop.io.BytesWritable.setSize(BytesWritable.java:76)
        at org.apache.hadoop.io.BytesWritable.readFields(BytesWritable.java:131)
        at org.apache.hadoop.mapred.JobClient$RawSplit.readFields(JobClient.java:797)
        at org.apache.hadoop.mapred.JobClient.readSplitFile(JobClient.java:863)
        at org.apache.hadoop.mapred.JobInProgress.initTasks(JobInProgress.java:308)
        at org.apache.hadoop.mapred.JobTracker$JobInitThread.run(JobTracker.java:418)
        at java.lang.Thread.run(Thread.java:619)

>From then on JobTracker became unresponsive, running GC at full speed.

Unfortunately, for some reason, I could not find a heap dump file.

Typical GCtimestamp output (it looks as if from a certain point on full GC is running repeatedly
with hardly any gain)


33015.775: [GC [PSYoungGen: 154229K->55555K(188352K)] 2344017K->2254659K(2463936K),
0.0688110 secs]
33027.318: [GC [PSYoungGen: 149123K->54221K(189632K)] 2348227K->2259655K(2465216K),
0.0603560 secs]
33046.658: [GC [PSYoungGen: 149069K->18692K(189632K)] 2354503K->2259525K(2465216K),
0.0683130 secs]
33056.766: [GC [PSYoungGen: 113537K->20288K(189632K)] 2354370K->2269026K(2465216K),
0.0415790 secs]
33056.808: [Full GC [PSYoungGen: 20288K->0K(189632K)] [PSOldGen: 2248737K->2268912K(2275584K)]
2269026K->2268
912K(2465216K) [PSPermGen: 11448K->11448K(16384K)], 1.7332610 secs]
33081.667: [Full GC [PSYoungGen: 94848K->0K(189632K)] [PSOldGen: 2268912K->2272832K(2275584K)]
2363760K->2272
832K(2465216K) [PSPermGen: 11448K->11448K(16384K)], 1.7537480 secs]
33096.646: [Full GC [PSYoungGen: 94848K->0K(189632K)] [PSOldGen: 2272832K->2262529K(2275584K)]
2367680K->2262
529K(2465216K) [PSPermGen: 11448K->11443K(16384K)], 3.2210170 secs]
33120.150: [Full GC [PSYoungGen: 94848K->0K(189632K)] [PSOldGen: 2262529K->2267044K(2275584K)]
2357377K->2267
044K(2465216K) [PSPermGen: 11443K->11443K(16384K)], 1.7487610 secs]
33136.949: [Full GC [PSYoungGen: 94848K->0K(189632K)] [PSOldGen: 2267044K->2272689K(2275584K)]
23618


> JobTracker running out of heap space
> ------------------------------------
>
>                 Key: HADOOP-3670
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3670
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.17.0
>            Reporter: Christian Kunz
>
> The JobTracker on our 0.17.0 installation runs out of heap space rather quickly, with
less than 100 jobs (at one time even after just 16 jobs).
> Running in 64-bit mode with larger heap space does not help -- it will use up all available
RAM.
> 2008-06-28 05:17:06,661 INFO org.apache.hadoop.ipc.Server: IPC Server handler 62 on 9020,
call he
> artbeat(org.apache.hadoop.mapred.TaskTrackerStatus@6f81c6, false, true, 17384) from xxx.xxx.xxx.xxx
> :51802: error: java.io.IOException: java.lang.OutOfMemoryError: GC overhead limit exceeded
> java.io.IOException: java.lang.OutOfMemoryError: GC overhead limit exceeded

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message