hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marcos Luis Ortiz Valmaseda <marcosluis2...@googlemail.com>
Subject Re: heap size problem durning mapreduce
Date Mon, 28 Nov 2011 18:41:30 GMT
Of course, a 32 bits OS can handle only 4 GB of memory (2^32), for
that reason, It's getting that error.

I personally recommend:
- Decreasy the value of HADOOP_HEAPSIZE
- Use a 64 bits Unix/Linux OS

Regards

2011/11/28 Wayne Wan <bestirwiny@gmail.com>:
> oh! seems your setting is out of the size that OS(32-bit) can manage!
>
> 2011/11/13 Hoot Thompson <hoot@ptpnow.com>
>
>> I can't seem to get past this heapsize error.  Relevant settings are as
>> follows:
>>
>> # The maximum amount of heap to use, in MB. Default is 1000.
>>  export HADOOP_HEAPSIZE=10000
>>
>> <name>mapred.child.java.opts</name>
>> <value>-Xmx1048576m</value>
>> </property>
>>
>>
>>
>> 11/11/12 17:38:27 INFO hpc.Driver: Jar
>> Name: /home/hadoop/querry/HadoopTest.jar
>> 0 [main] INFO nccs.hpc.Driver  - Jar
>> Name: /home/hadoop/querry/HadoopTest.jar
>> 0 [main] INFO nccs.hpc.Driver  - Jar
>> Name: /home/hadoop/querry/HadoopTest.jar
>> 11/11/12 17:38:27 WARN mapred.JobClient: Use GenericOptionsParser for
>> parsing the arguments. Applications should implement Tool for the same.
>> 146 [main] WARN org.apache.hadoop.mapred.JobClient  - Use
>> GenericOptionsParser for parsing the arguments. Applications should
>> implement Tool for the same.
>> 146 [main] WARN org.apache.hadoop.mapred.JobClient  - Use
>> GenericOptionsParser for parsing the arguments. Applications should
>> implement Tool for the same.
>> 11/11/12 17:38:30 INFO input.FileInputFormat: Total input paths to
>> process : 2329
>> 3330 [main] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat
>> - Total input paths to process : 2329
>> 3330 [main] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat
>> - Total input paths to process : 2329
>> 11/11/12 17:38:34 INFO mapred.JobClient: Running job:
>> job_201111121210_0001
>> 7919 [main] INFO org.apache.hadoop.mapred.JobClient  - Running job:
>> job_201111121210_0001
>> 7919 [main] INFO org.apache.hadoop.mapred.JobClient  - Running job:
>> job_201111121210_0001
>> 11/11/12 17:38:35 INFO mapred.JobClient:  map 0% reduce 0%
>> 8922 [main] INFO org.apache.hadoop.mapred.JobClient  -  map 0% reduce 0%
>> 8922 [main] INFO org.apache.hadoop.mapred.JobClient  -  map 0% reduce 0%
>> 11/11/12 17:40:37 INFO mapred.JobClient:  map 1% reduce 0%
>> 130058 [main] INFO org.apache.hadoop.mapred.JobClient  -  map 1% reduce
>> 0%
>> 130058 [main] INFO org.apache.hadoop.mapred.JobClient  -  map 1% reduce
>> 0%
>> 11/11/12 17:42:17 INFO mapred.JobClient:  map 2% reduce 0%
>> 230162 [main] INFO org.apache.hadoop.mapred.JobClient  -  map 2% reduce
>> 0%
>> 230162 [main] INFO org.apache.hadoop.mapred.JobClient  -  map 2% reduce
>> 0%
>> 11/11/12 17:43:47 INFO mapred.JobClient:  map 3% reduce 0%
>> 320251 [main] INFO org.apache.hadoop.mapred.JobClient  -  map 3% reduce
>> 0%
>> 320251 [main] INFO org.apache.hadoop.mapred.JobClient  -  map 3% reduce
>> 0%
>> 11/11/12 17:45:26 INFO mapred.JobClient:  map 4% reduce 0%
>> 419346 [main] INFO org.apache.hadoop.mapred.JobClient  -  map 4% reduce
>> 0%
>> 419346 [main] INFO org.apache.hadoop.mapred.JobClient  -  map 4% reduce
>> 0%
>> 11/11/12 17:46:59 INFO mapred.JobClient:  map 5% reduce 0%
>> 512435 [main] INFO org.apache.hadoop.mapred.JobClient  -  map 5% reduce
>> 0%
>> 512435 [main] INFO org.apache.hadoop.mapred.JobClient  -  map 5% reduce
>> 0%
>> 11/11/12 17:48:26 INFO mapred.JobClient: Task Id :
>> attempt_201111121210_0001_r_000000_0, Status : FAILED
>> 599520 [main] INFO org.apache.hadoop.mapred.JobClient  - Task Id :
>> attempt_201111121210_0001_r_000000_0, Status : FAILED
>> 599520 [main] INFO org.apache.hadoop.mapred.JobClient  - Task Id :
>> attempt_201111121210_0001_r_000000_0, Status : FAILED
>> Error: java.lang.OutOfMemoryError: Java heap space
>>        at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
>> $MapOutputCopier.shuffleInMemory(ReduceTask.java:1685)
>>        at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
>> $MapOutputCopier.getMapOutput(ReduceTask.java:1545)
>>        at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
>> $MapOutputCopier.copyOutput(ReduceTask.java:1394)
>>        at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
>> $MapOutputCopier.run(ReduceTask.java:1326)
>>
>>
>>
>



-- 
Marcos Luis Ortíz Valmaseda
 Linux Infrastructure Engineer
 Linux User # 418229
 http://marcosluis2186.posterous.com
 http://www.linkedin.com/in/marcosluis2186
 Twitter: @marcosluis2186

Mime
View raw message