hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Amar Kamat <ama...@yahoo-inc.com>
Subject Re: about the length of Text
Date Mon, 25 Feb 2008 07:57:06 GMT
What is the heap size (see mapred.child.java.opts in your job.xml)? By the 
way what is the job configuration i.e what is the job type (sort etc), number of nodes 
etc. Can you provide more details? Anyways try increasing it which you can 
do it using
hadoop jar jar-file.jar program -Dmapred.child.java.opts=-Xmx512m <args>

On Mon, 25 Feb 2008, ma qiang wrote:

> Hi all;
>    A error happen in my program and system print as below:
>        java.lang.OutOfMemoryError: Java heap space
> 	at java.util.Arrays.copyOf(Arrays.java:2786)
> 	at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
> 	at java.io.DataOutputStream.write(DataOutputStream.java:90)
> 	at org.apache.hadoop.io.Text.write(Text.java:243)
> 	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:349)
> 	at MyTest.map(UserMinHashMap.java:43)
> 	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> 	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:192)
> 	at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1787)
> and I think the error happen here: output.collect(new
> Text(testKeyString),new Text(testValueString));
> I'm not sure why? I guess the reason is the length of testValueString
> is too large to construct Text or write into HD.   Who can tell why?
> Thanks very much
> Best Wishes!

View raw message