hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alex Kozlov <ale...@cloudera.com>
Subject Re: java.lang.OutOfMemoryError: Java heap space
Date Sat, 10 Jul 2010 19:30:46 GMT
Hi Shuja,

It looks like the OOM is happening in your code.  Are you running MapReduce
in a cluster?  If so, can you send the exact command line your code is
invoked with -- you can get it with a 'ps -Af | grep Nodemapper5.groovy'
command on one of the nodes which is running the task?

Thanks,

Alex K

On Sat, Jul 10, 2010 at 10:40 AM, Shuja Rehman <shujamughal@gmail.com>wrote:

> Hi All
>
> I am facing a hard problem. I am running a map reduce job using streaming
> but it fails and it gives the following error.
>
> Caught: java.lang.OutOfMemoryError: Java heap space
>        at Nodemapper5.parseXML(Nodemapper5.groovy:25)
>
> java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess
> failed with code 1
>        at
> org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:362)
>        at
> org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:572)
>
>        at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:136)
>        at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
>        at
> org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:36)
>        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
>
>        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
>        at org.apache.hadoop.mapred.Child.main(Child.java:170)
>
>
> I have increased the heap size in hadoop-env.sh and make it 2000M. Also I
> tell the job manually by following line.
>
> -D mapred.child.java.opts=-Xmx2000M \
>
> but it still gives the error. The same job runs fine if i run on shell
> using
> 1024M heap size like
>
> cat file.xml | /root/Nodemapper5.groovy
>
>
> Any clue?????????
>
> Thanks in advance.
>
> --
> Regards
> Shuja-ur-Rehman Baig
> _________________________________
> MS CS - School of Science and Engineering
> Lahore University of Management Sciences (LUMS)
> Sector U, DHA, Lahore, 54792, Pakistan
> Cell: +92 3214207445
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message