hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From haihong lu <ung3...@gmail.com>
Subject Re: GC overhead limit exceeded
Date Thu, 13 Mar 2014 02:16:56 GMT
Thanks a lot, the answer is helpful.


On Wed, Mar 12, 2014 at 2:20 PM, divye sheth <divs.sheth@gmail.com> wrote:

> Hi Haihong,
>
> Please check out the link below, I believe it should solve your problem.
>
>
> http://stackoverflow.com/questions/21005643/container-is-running-beyond-memory-limits
>
> Thanks
> Divye Sheth
>
>
> On Wed, Mar 12, 2014 at 11:33 AM, haihong lu <ung3210@gmail.com> wrote:
>
>> Thanks, even if i had added this parameter, but had no effect.
>>
>>
>> On Tue, Mar 11, 2014 at 12:11 PM, unmesha sreeveni <unmeshabiju@gmail.com
>> > wrote:
>>
>>> Try to increase the memory for datanode and see.This need to restart
>>> hadoop
>>> export HADOOP_DATANODE_OPTS="-Xmx10g"
>>> This will set the heap to 10gb
>>> You can also add this in start of hadoop-env.sh file
>>>
>>>
>>> On Tue, Mar 11, 2014 at 9:02 AM, haihong lu <ung3210@gmail.com> wrote:
>>>
>>>> i have tried both of the methods you side, but the problem still
>>>> exists. Thanks all the same. by the way, my hadoop version is 2.2.0, so the
>>>> parameter  "mapreduce.map.memory.mb =3072" added to mapred-site.xml
>>>> maybe has no effect. I have looked for this parameter in the document of
>>>> hadoop, but did not found it.
>>>>
>>>>
>>>> On Fri, Mar 7, 2014 at 4:57 PM, shashwat shriparv <
>>>> dwivedishashwat@gmail.com> wrote:
>>>>
>>>>> Check this out
>>>>>
>>>>>
>>>>> http://ask.gopivotal.com/hc/en-us/articles/201850408-Namenode-fails-with-java-lang-OutOfMemoryError-GC-overhead-limit-exceeded
>>>>>
>>>>>
>>>>>
>>>>> * Warm Regards_**∞_*
>>>>> * Shashwat Shriparv*
>>>>>  [image: http://www.linkedin.com/pub/shashwat-shriparv/19/214/2a9]<http://www.linkedin.com/pub/shashwat-shriparv/19/214/2a9>[image:
>>>>> https://twitter.com/shriparv] <https://twitter.com/shriparv>[image:
>>>>> https://www.facebook.com/shriparv] <https://www.facebook.com/shriparv>[image:
>>>>> http://google.com/+ShashwatShriparv]<http://google.com/+ShashwatShriparv>[image:
>>>>> http://www.youtube.com/user/sShriparv/videos]<http://www.youtube.com/user/sShriparv/videos>[image:
>>>>> http://profile.yahoo.com/SWXSTW3DVSDTF2HHSRM47AV6DI/]<shriparv@yahoo.com>
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Mar 7, 2014 at 12:04 PM, haihong lu <ung3210@gmail.com>
wrote:
>>>>>
>>>>>> Hi:
>>>>>>
>>>>>>      i have a problem when run Hibench with hadoop-2.2.0, the wrong
>>>>>> message list as below
>>>>>>
>>>>>> 14/03/07 13:54:53 INFO mapreduce.Job:  map 19% reduce 0%
>>>>>>  14/03/07 13:54:54 INFO mapreduce.Job:  map 21% reduce 0%
>>>>>> 14/03/07 14:00:26 INFO mapreduce.Job: Task Id :
>>>>>> attempt_1394160253524_0010_m_000020_0, Status : FAILED
>>>>>> Error: GC overhead limit exceeded
>>>>>> 14/03/07 14:00:27 INFO mapreduce.Job:  map 20% reduce 0%
>>>>>> 14/03/07 14:00:40 INFO mapreduce.Job: Task Id :
>>>>>> attempt_1394160253524_0010_m_000008_0, Status : FAILED
>>>>>> Error: GC overhead limit exceeded
>>>>>> 14/03/07 14:00:41 INFO mapreduce.Job:  map 19% reduce 0%
>>>>>> 14/03/07 14:00:59 INFO mapreduce.Job:  map 20% reduce 0%
>>>>>> 14/03/07 14:00:59 INFO mapreduce.Job: Task Id :
>>>>>> attempt_1394160253524_0010_m_000015_0, Status : FAILED
>>>>>> Error: GC overhead limit exceeded
>>>>>> 14/03/07 14:01:00 INFO mapreduce.Job:  map 19% reduce 0%
>>>>>> 14/03/07 14:01:03 INFO mapreduce.Job: Task Id :
>>>>>> attempt_1394160253524_0010_m_000023_0, Status : FAILED
>>>>>> Error: GC overhead limit exceeded
>>>>>> 14/03/07 14:01:11 INFO mapreduce.Job: Task Id :
>>>>>> attempt_1394160253524_0010_m_000026_0, Status : FAILED
>>>>>> Error: GC overhead limit exceeded
>>>>>> 14/03/07 14:01:35 INFO mapreduce.Job:  map 20% reduce 0%
>>>>>> 14/03/07 14:01:35 INFO mapreduce.Job: Task Id :
>>>>>> attempt_1394160253524_0010_m_000019_0, Status : FAILED
>>>>>> Error: GC overhead limit exceeded
>>>>>> 14/03/07 14:01:36 INFO mapreduce.Job:  map 19% reduce 0%
>>>>>> 14/03/07 14:01:43 INFO mapreduce.Job: Task Id :
>>>>>> attempt_1394160253524_0010_m_000007_0, Status : FAILED
>>>>>> Error: GC overhead limit exceeded
>>>>>> 14/03/07 14:02:00 INFO mapreduce.Job: Task Id :
>>>>>> attempt_1394160253524_0010_m_000000_0, Status : FAILED
>>>>>> Error: GC overhead limit exceeded
>>>>>> 14/03/07 14:02:01 INFO mapreduce.Job:  map 18% reduce 0%
>>>>>> 14/03/07 14:02:23 INFO mapreduce.Job: Task Id :
>>>>>> attempt_1394160253524_0010_m_000021_0, Status : FAILED
>>>>>> Error: Java heap space
>>>>>> 14/03/07 14:02:24 INFO mapreduce.Job:  map 17% reduce 0%
>>>>>> 14/03/07 14:02:31 INFO mapreduce.Job:  map 18% reduce 0%
>>>>>> 14/03/07 14:02:33 INFO mapreduce.Job: Task Id :
>>>>>> attempt_1394160253524_0010_m_000029_0, Status : FAILED
>>>>>> Error: GC overhead limit exceeded
>>>>>> 14/03/07 14:02:34 INFO mapreduce.Job:  map 17% reduce 0%
>>>>>> 14/03/07 14:02:38 INFO mapreduce.Job: Task Id :
>>>>>> attempt_1394160253524_0010_m_000010_0, Status : FAILED
>>>>>> Error: GC overhead limit exceeded
>>>>>> 14/03/07 14:02:41 INFO mapreduce.Job: Task Id :
>>>>>> attempt_1394160253524_0010_m_000018_0, Status : FAILED
>>>>>> Error: GC overhead limit exceeded
>>>>>> 14/03/07 14:02:43 INFO mapreduce.Job: Task Id :
>>>>>> attempt_1394160253524_0010_m_000014_0, Status : FAILED
>>>>>> Error: GC overhead limit exceeded
>>>>>> 14/03/07 14:02:47 INFO mapreduce.Job: Task Id :
>>>>>> attempt_1394160253524_0010_m_000028_0, Status : FAILED
>>>>>> Error: Java heap space
>>>>>> 14/03/07 14:02:50 INFO mapreduce.Job: Task Id :
>>>>>> attempt_1394160253524_0010_m_000002_0, Status : FAILED
>>>>>> Error: GC overhead limit exceeded
>>>>>> 14/03/07 14:02:51 INFO mapreduce.Job:  map 16% reduce 0%
>>>>>> 14/03/07 14:02:51 INFO mapreduce.Job: Task Id :
>>>>>> attempt_1394160253524_0010_m_000005_0, Status : FAILED
>>>>>> Error: GC overhead limit exceeded
>>>>>> 14/03/07 14:02:52 INFO mapreduce.Job:  map 15% reduce 0%
>>>>>> 14/03/07 14:02:55 INFO mapreduce.Job: Task Id :
>>>>>> attempt_1394160253524_0010_m_000006_0, Status : FAILED
>>>>>> Error: GC overhead limit exceeded
>>>>>> 14/03/07 14:02:57 INFO mapreduce.Job: Task Id :
>>>>>> attempt_1394160253524_0010_m_000027_0, Status : FAILED
>>>>>> Error: GC overhead limit exceeded
>>>>>> 14/03/07 14:02:58 INFO mapreduce.Job:  map 14% reduce 0%
>>>>>> 14/03/07 14:03:04 INFO mapreduce.Job: Task Id :
>>>>>> attempt_1394160253524_0010_m_000009_0, Status : FAILED
>>>>>> Error: GC overhead limit exceeded
>>>>>> 14/03/07 14:03:05 INFO mapreduce.Job: Task Id :
>>>>>> attempt_1394160253524_0010_m_000017_0, Status : FAILED
>>>>>> Error: GC overhead limit exceeded
>>>>>> 14/03/07 14:03:05 INFO mapreduce.Job: Task Id :
>>>>>> attempt_1394160253524_0010_m_000022_0, Status : FAILED
>>>>>> Error: GC overhead limit exceeded
>>>>>> 14/03/07 14:03:06 INFO mapreduce.Job:  map 12% reduce 0%
>>>>>> 14/03/07 14:03:10 INFO mapreduce.Job: Task Id :
>>>>>> attempt_1394160253524_0010_m_000001_0, Status : FAILED
>>>>>> Error: GC overhead limit exceeded
>>>>>> 14/03/07 14:03:11 INFO mapreduce.Job:  map 13% reduce 0%
>>>>>> 14/03/07 14:03:11 INFO mapreduce.Job: Task Id :
>>>>>> attempt_1394160253524_0010_m_000024_0, Status : FAILED
>>>>>>
>>>>>> and then i add a parameter "mapred.child.java.opts" to the file
>>>>>> "mapred-site.xml",
>>>>>>   <property>
>>>>>>         <name>mapred.child.java.opts</name>
>>>>>>         <value>-Xmx1024m</value>
>>>>>>   </property>
>>>>>> then another error occurs as below
>>>>>>
>>>>>> 14/03/07 11:21:51 INFO mapreduce.Job:  map 0% reduce 0%
>>>>>> 14/03/07 11:21:59 INFO mapreduce.Job: Task Id :
>>>>>> attempt_1394160253524_0003_m_000002_0, Status : FAILED
>>>>>> Container
>>>>>> [pid=5592,containerID=container_1394160253524_0003_01_000004] is
running
>>>>>> beyond virtual memory limits. Current usage: 112.6 MB of 1 GB physical
>>>>>> memory used; 2.7 GB of
>>>>>>
>>>>>> 2.1 GB virtual memory used. Killing container.
>>>>>> Dump of the process-tree for container_1394160253524_0003_01_000004
:
>>>>>> |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS)
>>>>>> SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
>>>>>>  |- 5598 5592 5592 5592 (java) 563 14 2778632192 28520
>>>>>> /usr/java/jdk1.7.0_45/bin/java -Djava.net.preferIPv4Stack=true
>>>>>> -Dhadoop.metrics.log.level=WARN -Xmx2048m -
>>>>>>
>>>>>> Djava.io.tmpdir=/home/hadoop/tmp/nm-local-dir/usercache/hadoop/appcache/application_1394160253524_0003/container_1394160253524_0003_01_000004/tmp
>>>>>> -
>>>>>>
>>>>>> Dlog4j.configuration=container-log4j.properties
>>>>>> -Dyarn.app.container.log.dir=/var/log/hadoop/yarn/userlogs/application_1394160253524_0003/container_1394160253524_0003_01_000004
>>>>>>
>>>>>> -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA
>>>>>> org.apache.hadoop.mapred.YarnChild 10.239.44.34 46837
>>>>>> attempt_1394160253524_0003_m_000002_0 4
>>>>>> |- 5592 4562 5592 5592 (bash) 0 0 108650496 300 /bin/bash -c
>>>>>> /usr/java/jdk1.7.0_45/bin/java -Djava.net.preferIPv4Stack=true
>>>>>> -Dhadoop.metrics.log.level=WARN  -Xmx2048m -
>>>>>>
>>>>>> Djava.io.tmpdir=/home/hadoop/tmp/nm-local-dir/usercache/hadoop/appcache/application_1394160253524_0003/container_1394160253524_0003_01_000004/tmp
>>>>>> -
>>>>>>
>>>>>> Dlog4j.configuration=container-log4j.properties
>>>>>> -Dyarn.app.container.log.dir=/var/log/hadoop/yarn/userlogs/application_1394160253524_0003/container_1394160253524_0003_01_000004
>>>>>>
>>>>>> -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA
>>>>>> org.apache.hadoop.mapred.YarnChild 10.239.44.34 46837
>>>>>> attempt_1394160253524_0003_m_000002_0 4
>>>>>>
>>>>>>
>>>>>> 1>/var/log/hadoop/yarn/userlogs/application_1394160253524_0003/container_1394160253524_0003_01_000004/stdout
>>>>>>
>>>>>> 2>/var/log/hadoop/yarn/userlogs/application_1394160253524_0003/container_1394160253524_0003_01_000004/stderr
>>>>>>
>>>>>>
>>>>>> Container killed on request. Exit code is 143
>>>>>> 14/03/07 11:22:02 INFO mapreduce.Job: Task Id :
>>>>>> attempt_1394160253524_0003_m_000001_0, Status : FAILED
>>>>>> Container
>>>>>> [pid=5182,containerID=container_1394160253524_0003_01_000003] is
running
>>>>>> beyond virtual memory limits. Current usage: 118.5 MB of 1 GB physical
>>>>>> memory used; 2.7 GB of
>>>>>>
>>>>>> 2.1 GB virtual memory used. Killing container.
>>>>>> Dump of the process-tree for container_1394160253524_0003_01_000003
:
>>>>>> |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS)
>>>>>> SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
>>>>>>  |- 5182 4313 5182 5182 (bash) 0 0 108650496 303 /bin/bash -c
>>>>>> /usr/java/jdk1.7.0_45/bin/java -Djava.net.preferIPv4Stack=true
>>>>>> -Dhadoop.metrics.log.level=WARN  -Xmx2048m -
>>>>>>
>>>>>> Djava.io.tmpdir=/home/hadoop/tmp/nm-local-dir/usercache/hadoop/appcache/application_1394160253524_0003/container_1394160253524_0003_01_000003/tmp
>>>>>> -
>>>>>>
>>>>>> Dlog4j.configuration=container-log4j.properties
>>>>>> -Dyarn.app.container.log.dir=/var/log/hadoop/yarn/userlogs/application_1394160253524_0003/container_1394160253524_0003_01_000003
>>>>>>
>>>>>> -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA
>>>>>> org.apache.hadoop.mapred.YarnChild 10.239.44.34 46837
>>>>>> attempt_1394160253524_0003_m_000001_0 3
>>>>>>
>>>>>>
>>>>>> 1>/var/log/hadoop/yarn/userlogs/application_1394160253524_0003/container_1394160253524_0003_01_000003/stdout
>>>>>>
>>>>>> 2>/var/log/hadoop/yarn/userlogs/application_1394160253524_0003/container_1394160253524_0003_01_000003/stderr
>>>>>>
>>>>>> |- 5187 5182 5182 5182 (java) 616 19 2783928320 30028
>>>>>> /usr/java/jdk1.7.0_45/bin/java -Djava.net.preferIPv4Stack=true
>>>>>> -Dhadoop.metrics.log.level=WARN -Xmx2048m -
>>>>>>
>>>>>> Djava.io.tmpdir=/home/hadoop/tmp/nm-local-dir/usercache/hadoop/appcache/application_1394160253524_0003/container_1394160253524_0003_01_000003/tmp
>>>>>> -
>>>>>>
>>>>>> Dlog4j.configuration=container-log4j.properties
>>>>>> -Dyarn.app.container.log.dir=/var/log/hadoop/yarn/userlogs/application_1394160253524_0003/container_1394160253524_0003_01_000003
>>>>>>
>>>>>> -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA
>>>>>> org.apache.hadoop.mapred.YarnChild 10.239.44.34 46837
>>>>>> attempt_1394160253524_0003_m_000001_0 3
>>>>>>
>>>>>> Container killed on request. Exit code is 143
>>>>>>
>>>>>> at last, the task failed.
>>>>>> Thanks for any help!
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> *Thanks & Regards*
>>>
>>> Unmesha Sreeveni U.B
>>> Junior Developer
>>>
>>> http://www.unmeshasreeveni.blogspot.in/
>>>
>>>
>>>
>>
>

Mime
View raw message