hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <ha...@cloudera.com>
Subject Re: heap size problem durning mapreduce
Date Mon, 28 Nov 2011 12:21:11 GMT
Hoot,

Your settings of 10 GB per child process, and 10 GB per daemon process
do not make sense.

Take a read at these slides, will help you configure your heap setup
properly, among other things:
http://www.slideshare.net/cloudera/hadoop-troubleshooting-101-kate-ting-cloudera

On Sun, Nov 13, 2011 at 4:44 AM, Hoot Thompson <hoot@ptpnow.com> wrote:
> I can't seem to get past this heapsize error.  Relevant settings are as
> follows:
>
> # The maximum amount of heap to use, in MB. Default is 1000.
>  export HADOOP_HEAPSIZE=10000
>
> <name>mapred.child.java.opts</name>
> <value>-Xmx1048576m</value>
> </property>
>
>
>
> 11/11/12 17:38:27 INFO hpc.Driver: Jar
> Name: /home/hadoop/querry/HadoopTest.jar
> 0 [main] INFO nccs.hpc.Driver  - Jar
> Name: /home/hadoop/querry/HadoopTest.jar
> 0 [main] INFO nccs.hpc.Driver  - Jar
> Name: /home/hadoop/querry/HadoopTest.jar
> 11/11/12 17:38:27 WARN mapred.JobClient: Use GenericOptionsParser for
> parsing the arguments. Applications should implement Tool for the same.
> 146 [main] WARN org.apache.hadoop.mapred.JobClient  - Use
> GenericOptionsParser for parsing the arguments. Applications should
> implement Tool for the same.
> 146 [main] WARN org.apache.hadoop.mapred.JobClient  - Use
> GenericOptionsParser for parsing the arguments. Applications should
> implement Tool for the same.
> 11/11/12 17:38:30 INFO input.FileInputFormat: Total input paths to
> process : 2329
> 3330 [main] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat
> - Total input paths to process : 2329
> 3330 [main] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat
> - Total input paths to process : 2329
> 11/11/12 17:38:34 INFO mapred.JobClient: Running job:
> job_201111121210_0001
> 7919 [main] INFO org.apache.hadoop.mapred.JobClient  - Running job:
> job_201111121210_0001
> 7919 [main] INFO org.apache.hadoop.mapred.JobClient  - Running job:
> job_201111121210_0001
> 11/11/12 17:38:35 INFO mapred.JobClient:  map 0% reduce 0%
> 8922 [main] INFO org.apache.hadoop.mapred.JobClient  -  map 0% reduce 0%
> 8922 [main] INFO org.apache.hadoop.mapred.JobClient  -  map 0% reduce 0%
> 11/11/12 17:40:37 INFO mapred.JobClient:  map 1% reduce 0%
> 130058 [main] INFO org.apache.hadoop.mapred.JobClient  -  map 1% reduce
> 0%
> 130058 [main] INFO org.apache.hadoop.mapred.JobClient  -  map 1% reduce
> 0%
> 11/11/12 17:42:17 INFO mapred.JobClient:  map 2% reduce 0%
> 230162 [main] INFO org.apache.hadoop.mapred.JobClient  -  map 2% reduce
> 0%
> 230162 [main] INFO org.apache.hadoop.mapred.JobClient  -  map 2% reduce
> 0%
> 11/11/12 17:43:47 INFO mapred.JobClient:  map 3% reduce 0%
> 320251 [main] INFO org.apache.hadoop.mapred.JobClient  -  map 3% reduce
> 0%
> 320251 [main] INFO org.apache.hadoop.mapred.JobClient  -  map 3% reduce
> 0%
> 11/11/12 17:45:26 INFO mapred.JobClient:  map 4% reduce 0%
> 419346 [main] INFO org.apache.hadoop.mapred.JobClient  -  map 4% reduce
> 0%
> 419346 [main] INFO org.apache.hadoop.mapred.JobClient  -  map 4% reduce
> 0%
> 11/11/12 17:46:59 INFO mapred.JobClient:  map 5% reduce 0%
> 512435 [main] INFO org.apache.hadoop.mapred.JobClient  -  map 5% reduce
> 0%
> 512435 [main] INFO org.apache.hadoop.mapred.JobClient  -  map 5% reduce
> 0%
> 11/11/12 17:48:26 INFO mapred.JobClient: Task Id :
> attempt_201111121210_0001_r_000000_0, Status : FAILED
> 599520 [main] INFO org.apache.hadoop.mapred.JobClient  - Task Id :
> attempt_201111121210_0001_r_000000_0, Status : FAILED
> 599520 [main] INFO org.apache.hadoop.mapred.JobClient  - Task Id :
> attempt_201111121210_0001_r_000000_0, Status : FAILED
> Error: java.lang.OutOfMemoryError: Java heap space
>        at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
> $MapOutputCopier.shuffleInMemory(ReduceTask.java:1685)
>        at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
> $MapOutputCopier.getMapOutput(ReduceTask.java:1545)
>        at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
> $MapOutputCopier.copyOutput(ReduceTask.java:1394)
>        at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
> $MapOutputCopier.run(ReduceTask.java:1326)
>
>
>



-- 
Harsh J

Mime
View raw message