Thanks Thomas.

Anohter question. I have no idea what is "Failed to merge in memory". Does the 'merge' is the shuffle phase= in reducer side? Why it is in memory?

Except the t= wo methods(increase reducer number and increase heap size), is there= any other alternatives to fix this issue?

= Thanks a lot.

=0A

leiwangouc@gmail.com

=0A

&n= bsp;
From: Thomas Bentsen
Date: = ;2014-04-15 21:53
To: user
Subject: Re: memoryjava.la= ng.OutOfMemoryError related with number of reducer?
=
When you increase the number of reducers they each have less to work<= /div>=0A
with provided the data is distributed evenly between them - i= n this case
=0A
about one third of the original work.
=0AIt is eessentially the same thing as increasing the heap size - it's=0A
just distributed between more reducers.
=0A
=0A
/th
=0A

=0A

=0A
=0A
On Tue, 2014-04-15 at 20:41 +0800, leiwangouc@gmail.com wrote:=
=0A
> I can fix this by changing heap size.
=0A
>= But what confuse me is that when i change the reducer number from 24=0A
> to 84, there's no this error.
=0A
>
=0A>
=0A
> Any insight on this?
=0A
>
=0A=
>
=0A
> Thanks
=0A
> Lei
=0A
>= ; Failed to merge in memoryjava.lang.OutOfMemoryError: Java heap space=0A
> at java.util.Arrays.copyOf(Arrays.java:2786)
=0A
= > at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94= )
=0A
> at java.io.DataOutputStream.write(DataOutputStream.ja= va:90)
=0A
> at java.io.DataOutputStream.writeUTF(DataOutputS= tream.java:384)
=0A
> at java.io.DataOutputStream.writeUTF(Da= taOutputStream.java:306)
=0A
> at org.apache.pig.data.utils.S= edesHelper.writeChararray(SedesHelper.java:66)
=0A
> at org.a= pache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:543)
=0A> at org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.jav= a:435)
=0A
> at org.apache.pig.data.utils.SedesHelper.writeGe= nericTuple(SedesHelper.java:135)
=0A
> at org.apache.pig.data= .BinInterSedes.writeTuple(BinInterSedes.java:613)
=0A
> at or= g.apache.pig.data.BinInterSedes.writeBag(BinInterSedes.java:604)
=0A<= div>> at org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.ja= va:447)
=0A
> at org.apache.pig.data.BinInterSedes.writeDatum= (BinInterSedes.java:435)
=0A
> at org.apache.pig.data.utils.S= edesHelper.writeGenericTuple(SedesHelper.java:135)
=0A
> at o= rg.apache.pig.data.BinInterSedes.writeTuple(BinInterSedes.java:613)
= =0A
> at org.apache.pig.data.BinInterSedes.writeDatum(BinInterSede= s.java:443)
=0A
> at org.apache.pig.data.BinInterSedes.writeD= atum(BinInterSedes.java:435)
=0A
> at org.apache.pig.data.uti= ls.SedesHelper.writeGenericTuple(SedesHelper.java:135)
=0A
> = at org.apache.pig.data.BinInterSedes.writeTuple(BinInterSedes.java:613)=0A
> at org.apache.pig.data.BinInterSedes.writeDatum(BinInterS= edes.java:443)
=0A
> at org.apache.pig.data.BinSedesTuple.wri= te(BinSedesTuple.java:41)
=0A
> at org.apache.pig.impl.io.Pig= NullableWritable.write(PigNullableWritable.java:123)
=0A
> at= org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.= serialize(WritableSerialization.java:100)
=0A
> at org.apache= .hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(W= ritableSerialization.java:84)
=0A
> at org.apache.hadoop.mapr= ed.IFile$Writer.append(IFile.java:188)
=0A
> at org.apache.ha= doop.mapred.Task$CombineOutputCollector.collect(Task.java:1145)
=0A> at org.apache.hadoop.mapred.Task$NewCombinerRunner$OutputConverte= r.write(Task.java:1456)
=0A
> at org.apache.hadoop.mapreduce.= task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:85)<= /div>=0A
> at org.apache.hadoop.mapreduce.lib.reduce.WrappedReduce= r$Context.write(WrappedReducer.java:99)
=0A
> at org.apache.p= ig.backend.hadoop.executionengine.mapReduceLayer.PigCombiner$Combine.proce= ssOnePackageOutput(PigCombiner.java:201)
=0A
> at org.apache.= pig.backend.hadoop.executionengine.mapReduceLayer.PigCombiner$Combine.redu= ce(PigCombiner.java:163)
=0A
> at org.apache.pig.backend.hado= op.executionengine.mapReduceLayer.PigCombiner$Combine.reduce(PigCombiner.j= ava:51)
=0A
>
=0A
> _____________________________= _________________________________________
=0A
> leiwangouc@gma= il.com
=0A

=0A

=0A

= =0A ------=_001_NextPart715541364524_=------