hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hoot Thompson <h...@ptpnow.com>
Subject Mapreduce heap size error
Date Sat, 12 Nov 2011 23:32:05 GMT


>> 
>>> CanĀ¹t seem to get past this heap size error, any ideas where to look? Below
>>> are my heap size settings, at least the ones I attempted to increase.
>>> 
>>> Thanks in advance for any thoughts.
>>> 
>>> 
>>> # The maximum amount of heap to use, in MB. Default is 1000.
>>>  export HADOOP_HEAPSIZE=10000
>>> 
>>> 
>>> 
>>> <name>mapred.child.java.opts</name>
>>> <value>-Xmx1048576m</value>
>>> </property>
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 11/11/12 17:38:27 INFO hpc.Driver: Jar Name:
>>> /home/hadoop/querry/HadoopTest.jar
>>> 0 [main] INFO nccs.hpc.Driver  - Jar Name:
>>> /home/hadoop/querry/HadoopTest.jar
>>> 0 [main] INFO nccs.hpc.Driver  - Jar Name:
>>> /home/hadoop/querry/HadoopTest.jar
>>> 11/11/12 17:38:27 WARN mapred.JobClient: Use GenericOptionsParser for
>>> parsing the arguments. Applications should implement Tool for the same.
>>> 146 [main] WARN org.apache.hadoop.mapred.JobClient  - Use
>>> GenericOptionsParser for parsing the arguments. Applications should
>>> implement Tool for the same.
>>> 146 [main] WARN org.apache.hadoop.mapred.JobClient  - Use
>>> GenericOptionsParser for parsing the arguments. Applications should
>>> implement Tool for the same.
>>> 11/11/12 17:38:30 INFO input.FileInputFormat: Total input paths to process :
>>> 2329
>>> 3330 [main] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat  -
>>> Total input paths to process : 2329
>>> 3330 [main] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat  -
>>> Total input paths to process : 2329
>>> 11/11/12 17:38:34 INFO mapred.JobClient: Running job: job_201111121210_0001
>>> 7919 [main] INFO org.apache.hadoop.mapred.JobClient  - Running job:
>>> job_201111121210_0001
>>> 7919 [main] INFO org.apache.hadoop.mapred.JobClient  - Running job:
>>> job_201111121210_0001
>>> 11/11/12 17:38:35 INFO mapred.JobClient:  map 0% reduce 0%
>>> 8922 [main] INFO org.apache.hadoop.mapred.JobClient  -  map 0% reduce 0%
>>> 8922 [main] INFO org.apache.hadoop.mapred.JobClient  -  map 0% reduce 0%
>>> 11/11/12 17:40:37 INFO mapred.JobClient:  map 1% reduce 0%
>>> 130058 [main] INFO org.apache.hadoop.mapred.JobClient  -  map 1% reduce 0%
>>> 130058 [main] INFO org.apache.hadoop.mapred.JobClient  -  map 1% reduce 0%
>>> 11/11/12 17:42:17 INFO mapred.JobClient:  map 2% reduce 0%
>>> 230162 [main] INFO org.apache.hadoop.mapred.JobClient  -  map 2% reduce 0%
>>> 230162 [main] INFO org.apache.hadoop.mapred.JobClient  -  map 2% reduce 0%
>>> 11/11/12 17:43:47 INFO mapred.JobClient:  map 3% reduce 0%
>>> 320251 [main] INFO org.apache.hadoop.mapred.JobClient  -  map 3% reduce 0%
>>> 320251 [main] INFO org.apache.hadoop.mapred.JobClient  -  map 3% reduce 0%
>>> 11/11/12 17:45:26 INFO mapred.JobClient:  map 4% reduce 0%
>>> 419346 [main] INFO org.apache.hadoop.mapred.JobClient  -  map 4% reduce 0%
>>> 419346 [main] INFO org.apache.hadoop.mapred.JobClient  -  map 4% reduce 0%
>>> 11/11/12 17:46:59 INFO mapred.JobClient:  map 5% reduce 0%
>>> 512435 [main] INFO org.apache.hadoop.mapred.JobClient  -  map 5% reduce 0%
>>> 512435 [main] INFO org.apache.hadoop.mapred.JobClient  -  map 5% reduce 0%
>>> 11/11/12 17:48:26 INFO mapred.JobClient: Task Id :
>>> attempt_201111121210_0001_r_000000_0, Status : FAILED
>>> 599520 [main] INFO org.apache.hadoop.mapred.JobClient  - Task Id :
>>> attempt_201111121210_0001_r_000000_0, Status : FAILED
>>> 599520 [main] INFO org.apache.hadoop.mapred.JobClient  - Task Id :
>>> attempt_201111121210_0001_r_000000_0, Status : FAILED
>>> Error: java.lang.OutOfMemoryError: Java heap space
>>>     at 
>>> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.shuffleInMe
>>> mory(ReduceTask.java:1685)
>>>     at 
>>> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutpu
>>> t(ReduceTask.java:1545)
>>>     at 
>>> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(
>>> ReduceTask.java:1394)
>>>     at 
>>> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceT
>>> ask.java:1326)
>>> 


Mime
View raw message