hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "leiwangouc@gmail.com" <leiwang...@gmail.com>
Subject memoryjava.lang.OutOfMemoryError related with number of reducer?
Date Tue, 15 Apr 2014 12:41:20 GMT
I can fix this by changing heap size.
But what confuse me is that when i change the reducer number from 24 to 84, there's no this
error.

Any insight on this?

Thanks
Lei
Failed to merge in memoryjava.lang.OutOfMemoryError: Java heap space
	at java.util.Arrays.copyOf(Arrays.java:2786)
	at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
	at java.io.DataOutputStream.write(DataOutputStream.java:90)
	at java.io.DataOutputStream.writeUTF(DataOutputStream.java:384)
	at java.io.DataOutputStream.writeUTF(DataOutputStream.java:306)
	at org.apache.pig.data.utils.SedesHelper.writeChararray(SedesHelper.java:66)
	at org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:543)
	at org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:435)
	at org.apache.pig.data.utils.SedesHelper.writeGenericTuple(SedesHelper.java:135)
	at org.apache.pig.data.BinInterSedes.writeTuple(BinInterSedes.java:613)
	at org.apache.pig.data.BinInterSedes.writeBag(BinInterSedes.java:604)
	at org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:447)
	at org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:435)
	at org.apache.pig.data.utils.SedesHelper.writeGenericTuple(SedesHelper.java:135)
	at org.apache.pig.data.BinInterSedes.writeTuple(BinInterSedes.java:613)
	at org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:443)
	at org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:435)
	at org.apache.pig.data.utils.SedesHelper.writeGenericTuple(SedesHelper.java:135)
	at org.apache.pig.data.BinInterSedes.writeTuple(BinInterSedes.java:613)
	at org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:443)
	at org.apache.pig.data.BinSedesTuple.write(BinSedesTuple.java:41)
	at org.apache.pig.impl.io.PigNullableWritable.write(PigNullableWritable.java:123)
	at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:100)
	at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:84)
	at org.apache.hadoop.mapred.IFile$Writer.append(IFile.java:188)
	at org.apache.hadoop.mapred.Task$CombineOutputCollector.collect(Task.java:1145)
	at org.apache.hadoop.mapred.Task$NewCombinerRunner$OutputConverter.write(Task.java:1456)
	at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:85)
	at org.apache.hadoop.mapreduce.lib.reduce.WrappedReducer$Context.write(WrappedReducer.java:99)
	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigCombiner$Combine.processOnePackageOutput(PigCombiner.java:201)
	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigCombiner$Combine.reduce(PigCombiner.java:163)
	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigCombiner$Combine.reduce(PigCombiner.java:51)


leiwangouc@gmail.com
Mime
View raw message