hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "XiaopengLi (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-6447) reduce shuffle throws "java.lang.OutOfMemoryError: Java heap space"
Date Wed, 12 Aug 2015 03:32:46 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-6447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14692856#comment-14692856
] 

XiaopengLi commented on MAPREDUCE-6447:
---------------------------------------

Hi,shuzhangyao.I have shared the same experiences. I ever thought about  your method .This
method is mandatory to limit within the memoryLimt.  It maybe is effective. But I think that
the original code  is  reasonable .  We take the default parameter to calculate it.  maxSingleShuffleLinit
= memorylimit  * 0.25. The number of Fetcher is 5. memorylimit =  Runtime.getRuntime().maxMemory()
* 0.7 . While all fetcher is working, 5*0.25*0.7 <1,in theory,it does not occur the OutOfMemory
of Java heap. Even though we do not add the code "usedMemory +requestedSize)< memoryLimit"
,in theory ,it should not occur this phenomenon of Outof Memory. We can talk about this problem.
whether It be casued by Jvm and  unreasonable allocation of memory in  special data input
?



> reduce shuffle throws "java.lang.OutOfMemoryError: Java heap space"
> -------------------------------------------------------------------
>
>                 Key: MAPREDUCE-6447
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6447
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>    Affects Versions: 2.5.0, 2.6.0, 2.5.1, 2.7.1
>            Reporter: shuzhangyao
>            Assignee: shuzhangyao
>            Priority: Minor
>
> 2015-08-11 14:03:54,550 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running
child : org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in
fetcher#10
> 	at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:134)
> 	at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:376)
> 	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:415)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
> 	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
> Caused by: java.lang.OutOfMemoryError: Java heap space
> 	at org.apache.hadoop.io.BoundedByteArrayOutputStream.<init>(BoundedByteArrayOutputStream.java:56)
> 	at org.apache.hadoop.io.BoundedByteArrayOutputStream.<init>(BoundedByteArrayOutputStream.java:46)
> 	at org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput.<init>(InMemoryMapOutput.java:63)
> 	at org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.unconditionalReserve(MergeManagerImpl.java:303)
> 	at org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.reserve(MergeManagerImpl.java:293)
> 	at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyMapOutput(Fetcher.java:511)
> 	at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:329)
> 	at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:193)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message