hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mohamed Riadh Trad <Mohamed.t...@inria.fr>
Subject Re: Mapreduce heap size error
Date Tue, 15 Nov 2011 09:53:10 GMT
java -version 

to check if the java you are using is a 32 bits or 64 bits version.

If you are using a 32bits version you cannot allow more than 3,5 Gbyte for the heap.


Trad Mohamed Riadh, M.Sc, Ing.
PhD. student
INRIA-TELECOM PARISTECH - ENPC School of International Management

Office: 11-15
Phone: (33)-1 39 63 59 33
Fax: (33)-1 39 63 56 74
Email: riadh.trad@inria.fr
Home page: http://www-rocq.inria.fr/who/Mohamed.Trad/




Le 15 nov. 2011 à 00:39, Hoot Thompson a écrit :

> Still issues, around 2300 unique files....
> 
> 
> hadoop@lobster-nfs:~/querry$ hadoop jar HadoopTest.jar -D mapred.child.java.opts=-Xmx4096M
hdfs://lobster-nfs:9000/hadoop_fs/dfs/merra/seq_out /hadoop_fs/dfs/output/test_111114_r2.out
> 11/11/15 01:56:20 INFO hpc.Driver: Jar Name: /home/hadoop/querry/HadoopTest.jar
> 0 [main] INFO nccs.hpc.Driver  - Jar Name: /home/hadoop/querry/HadoopTest.jar
> 0 [main] INFO nccs.hpc.Driver  - Jar Name: /home/hadoop/querry/HadoopTest.jar
> 11/11/15 01:56:20 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments.
Applications should implement Tool for the same.
> 60 [main] WARN org.apache.hadoop.mapred.JobClient  - Use GenericOptionsParser for parsing
the arguments. Applications should implement Tool for the same.
> 60 [main] WARN org.apache.hadoop.mapred.JobClient  - Use GenericOptionsParser for parsing
the arguments. Applications should implement Tool for the same.
> 11/11/15 01:56:22 INFO input.FileInputFormat: Total input paths to process : 2329
> 2154 [main] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat  - Total input
paths to process : 2329
> 2154 [main] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat  - Total input
paths to process : 2329
> 11/11/15 01:56:24 INFO mapred.JobClient: Running job: job_201111150008_0003
> 3693 [main] INFO org.apache.hadoop.mapred.JobClient  - Running job: job_201111150008_0003
> 3693 [main] INFO org.apache.hadoop.mapred.JobClient  - Running job: job_201111150008_0003
> 11/11/15 01:56:25 INFO mapred.JobClient:  map 0% reduce 0%
> 4696 [main] INFO org.apache.hadoop.mapred.JobClient  -  map 0% reduce 0%
> 4696 [main] INFO org.apache.hadoop.mapred.JobClient  -  map 0% reduce 0%
> 11/11/15 01:58:03 INFO mapred.JobClient:  map 1% reduce 0%
> 102916 [main] INFO org.apache.hadoop.mapred.JobClient  -  map 1% reduce 0%
> 102916 [main] INFO org.apache.hadoop.mapred.JobClient  -  map 1% reduce 0%
> 11/11/15 01:59:07 INFO mapred.JobClient:  map 2% reduce 0%
> 167042 [main] INFO org.apache.hadoop.mapred.JobClient  -  map 2% reduce 0%
> 167042 [main] INFO org.apache.hadoop.mapred.JobClient  -  map 2% reduce 0%
> 11/11/15 02:00:12 INFO mapred.JobClient:  map 3% reduce 0%
> 232168 [main] INFO org.apache.hadoop.mapred.JobClient  -  map 3% reduce 0%
> 232168 [main] INFO org.apache.hadoop.mapred.JobClient  -  map 3% reduce 0%
> 11/11/15 02:01:18 INFO mapred.JobClient:  map 4% reduce 0%
> 298290 [main] INFO org.apache.hadoop.mapred.JobClient  -  map 4% reduce 0%
> 298290 [main] INFO org.apache.hadoop.mapred.JobClient  -  map 4% reduce 0%
> 11/11/15 02:02:21 INFO mapred.JobClient:  map 5% reduce 0%
> 361418 [main] INFO org.apache.hadoop.mapred.JobClient  -  map 5% reduce 0%
> 361418 [main] INFO org.apache.hadoop.mapred.JobClient  -  map 5% reduce 0%
> 11/11/15 02:02:51 INFO mapred.JobClient: Task Id : attempt_201111150008_0003_r_000000_0,
Status : FAILED
> 391477 [main] INFO org.apache.hadoop.mapred.JobClient  - Task Id : attempt_201111150008_0003_r_000000_0,
Status : FAILED
> 391477 [main] INFO org.apache.hadoop.mapred.JobClient  - Task Id : attempt_201111150008_0003_r_000000_0,
Status : FAILED
> Error: java.lang.OutOfMemoryError: Java heap space
>     at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.shuffleInMemory(ReduceTask.java:1685)
>     at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutput(ReduceTask.java:1545)
>     at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(ReduceTask.java:1394)
>     at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceTask.java:1326)
> 
> 
> 
> On 11/14/11 8:23 PM, "Mohamed Riadh Trad" <Mohamed.trad@inria.fr> wrote:
> 
>> -D mapred.child.java.opts=-Xmx4096M


Mime
View raw message