Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of leiwangouc@gmail.com designates
 209.85.192.174 as permitted sender)
Date: Tue, 15 Apr 2014 20:41:20 +0800
From: "leiwangouc@gmail.com" <leiwangouc@gmail.com>
To: user <user@pig.apache.org>,
	user <user@hadoop.apache.org>
Subject: memoryjava.lang.OutOfMemoryError related with number of reducer?
Mime-Version: 1.0
Message-ID: <2014041520410525592328@gmail.com>
Content-Type: multipart/alternative;
	boundary="----=_001_NextPart302674732371_=----"

This is a multi-part message in MIME format.

------=_001_NextPart302674732371_=----
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: base64

SSBjYW4gZml4IHRoaXMgYnkgY2hhbmdpbmcgaGVhcCBzaXplLg0KQnV0IHdoYXQgY29uZnVzZSBt
ZSBpcyB0aGF0IHdoZW4gaSBjaGFuZ2UgdGhlIHJlZHVjZXIgbnVtYmVyIGZyb20gMjQgdG8gODQs
IHRoZXJlJ3Mgbm8gdGhpcyBlcnJvci4NCg0KQW55IGluc2lnaHQgb24gdGhpcz8NCg0KVGhhbmtz
DQpMZWkNCkZhaWxlZCB0byBtZXJnZSBpbiBtZW1vcnlqYXZhLmxhbmcuT3V0T2ZNZW1vcnlFcnJv
cjogSmF2YSBoZWFwIHNwYWNlCglhdCBqYXZhLnV0aWwuQXJyYXlzLmNvcHlPZihBcnJheXMuamF2
YToyNzg2KQoJYXQgamF2YS5pby5CeXRlQXJyYXlPdXRwdXRTdHJlYW0ud3JpdGUoQnl0ZUFycmF5
T3V0cHV0U3RyZWFtLmphdmE6OTQpCglhdCBqYXZhLmlvLkRhdGFPdXRwdXRTdHJlYW0ud3JpdGUo
RGF0YU91dHB1dFN0cmVhbS5qYXZhOjkwKQoJYXQgamF2YS5pby5EYXRhT3V0cHV0U3RyZWFtLndy
aXRlVVRGKERhdGFPdXRwdXRTdHJlYW0uamF2YTozODQpCglhdCBqYXZhLmlvLkRhdGFPdXRwdXRT
dHJlYW0ud3JpdGVVVEYoRGF0YU91dHB1dFN0cmVhbS5qYXZhOjMwNikKCWF0IG9yZy5hcGFjaGUu
cGlnLmRhdGEudXRpbHMuU2VkZXNIZWxwZXIud3JpdGVDaGFyYXJyYXkoU2VkZXNIZWxwZXIuamF2
YTo2NikKCWF0IG9yZy5hcGFjaGUucGlnLmRhdGEuQmluSW50ZXJTZWRlcy53cml0ZURhdHVtKEJp
bkludGVyU2VkZXMuamF2YTo1NDMpCglhdCBvcmcuYXBhY2hlLnBpZy5kYXRhLkJpbkludGVyU2Vk
ZXMud3JpdGVEYXR1bShCaW5JbnRlclNlZGVzLmphdmE6NDM1KQoJYXQgb3JnLmFwYWNoZS5waWcu
ZGF0YS51dGlscy5TZWRlc0hlbHBlci53cml0ZUdlbmVyaWNUdXBsZShTZWRlc0hlbHBlci5qYXZh
OjEzNSkKCWF0IG9yZy5hcGFjaGUucGlnLmRhdGEuQmluSW50ZXJTZWRlcy53cml0ZVR1cGxlKEJp
bkludGVyU2VkZXMuamF2YTo2MTMpCglhdCBvcmcuYXBhY2hlLnBpZy5kYXRhLkJpbkludGVyU2Vk
ZXMud3JpdGVCYWcoQmluSW50ZXJTZWRlcy5qYXZhOjYwNCkKCWF0IG9yZy5hcGFjaGUucGlnLmRh
dGEuQmluSW50ZXJTZWRlcy53cml0ZURhdHVtKEJpbkludGVyU2VkZXMuamF2YTo0NDcpCglhdCBv
cmcuYXBhY2hlLnBpZy5kYXRhLkJpbkludGVyU2VkZXMud3JpdGVEYXR1bShCaW5JbnRlclNlZGVz
LmphdmE6NDM1KQoJYXQgb3JnLmFwYWNoZS5waWcuZGF0YS51dGlscy5TZWRlc0hlbHBlci53cml0
ZUdlbmVyaWNUdXBsZShTZWRlc0hlbHBlci5qYXZhOjEzNSkKCWF0IG9yZy5hcGFjaGUucGlnLmRh
dGEuQmluSW50ZXJTZWRlcy53cml0ZVR1cGxlKEJpbkludGVyU2VkZXMuamF2YTo2MTMpCglhdCBv
cmcuYXBhY2hlLnBpZy5kYXRhLkJpbkludGVyU2VkZXMud3JpdGVEYXR1bShCaW5JbnRlclNlZGVz
LmphdmE6NDQzKQoJYXQgb3JnLmFwYWNoZS5waWcuZGF0YS5CaW5JbnRlclNlZGVzLndyaXRlRGF0
dW0oQmluSW50ZXJTZWRlcy5qYXZhOjQzNSkKCWF0IG9yZy5hcGFjaGUucGlnLmRhdGEudXRpbHMu
U2VkZXNIZWxwZXIud3JpdGVHZW5lcmljVHVwbGUoU2VkZXNIZWxwZXIuamF2YToxMzUpCglhdCBv
cmcuYXBhY2hlLnBpZy5kYXRhLkJpbkludGVyU2VkZXMud3JpdGVUdXBsZShCaW5JbnRlclNlZGVz
LmphdmE6NjEzKQoJYXQgb3JnLmFwYWNoZS5waWcuZGF0YS5CaW5JbnRlclNlZGVzLndyaXRlRGF0
dW0oQmluSW50ZXJTZWRlcy5qYXZhOjQ0MykKCWF0IG9yZy5hcGFjaGUucGlnLmRhdGEuQmluU2Vk
ZXNUdXBsZS53cml0ZShCaW5TZWRlc1R1cGxlLmphdmE6NDEpCglhdCBvcmcuYXBhY2hlLnBpZy5p
bXBsLmlvLlBpZ051bGxhYmxlV3JpdGFibGUud3JpdGUoUGlnTnVsbGFibGVXcml0YWJsZS5qYXZh
OjEyMykKCWF0IG9yZy5hcGFjaGUuaGFkb29wLmlvLnNlcmlhbGl6ZXIuV3JpdGFibGVTZXJpYWxp
emF0aW9uJFdyaXRhYmxlU2VyaWFsaXplci5zZXJpYWxpemUoV3JpdGFibGVTZXJpYWxpemF0aW9u
LmphdmE6MTAwKQoJYXQgb3JnLmFwYWNoZS5oYWRvb3AuaW8uc2VyaWFsaXplci5Xcml0YWJsZVNl
cmlhbGl6YXRpb24kV3JpdGFibGVTZXJpYWxpemVyLnNlcmlhbGl6ZShXcml0YWJsZVNlcmlhbGl6
YXRpb24uamF2YTo4NCkKCWF0IG9yZy5hcGFjaGUuaGFkb29wLm1hcHJlZC5JRmlsZSRXcml0ZXIu
YXBwZW5kKElGaWxlLmphdmE6MTg4KQoJYXQgb3JnLmFwYWNoZS5oYWRvb3AubWFwcmVkLlRhc2sk
Q29tYmluZU91dHB1dENvbGxlY3Rvci5jb2xsZWN0KFRhc2suamF2YToxMTQ1KQoJYXQgb3JnLmFw
YWNoZS5oYWRvb3AubWFwcmVkLlRhc2skTmV3Q29tYmluZXJSdW5uZXIkT3V0cHV0Q29udmVydGVy
LndyaXRlKFRhc2suamF2YToxNDU2KQoJYXQgb3JnLmFwYWNoZS5oYWRvb3AubWFwcmVkdWNlLnRh
c2suVGFza0lucHV0T3V0cHV0Q29udGV4dEltcGwud3JpdGUoVGFza0lucHV0T3V0cHV0Q29udGV4
dEltcGwuamF2YTo4NSkKCWF0IG9yZy5hcGFjaGUuaGFkb29wLm1hcHJlZHVjZS5saWIucmVkdWNl
LldyYXBwZWRSZWR1Y2VyJENvbnRleHQud3JpdGUoV3JhcHBlZFJlZHVjZXIuamF2YTo5OSkKCWF0
IG9yZy5hcGFjaGUucGlnLmJhY2tlbmQuaGFkb29wLmV4ZWN1dGlvbmVuZ2luZS5tYXBSZWR1Y2VM
YXllci5QaWdDb21iaW5lciRDb21iaW5lLnByb2Nlc3NPbmVQYWNrYWdlT3V0cHV0KFBpZ0NvbWJp
bmVyLmphdmE6MjAxKQoJYXQgb3JnLmFwYWNoZS5waWcuYmFja2VuZC5oYWRvb3AuZXhlY3V0aW9u
ZW5naW5lLm1hcFJlZHVjZUxheWVyLlBpZ0NvbWJpbmVyJENvbWJpbmUucmVkdWNlKFBpZ0NvbWJp
bmVyLmphdmE6MTYzKQoJYXQgb3JnLmFwYWNoZS5waWcuYmFja2VuZC5oYWRvb3AuZXhlY3V0aW9u
ZW5naW5lLm1hcFJlZHVjZUxheWVyLlBpZ0NvbWJpbmVyJENvbWJpbmUucmVkdWNlKFBpZ0NvbWJp
bmVyLmphdmE6NTEpDQoNCg0KbGVpd2FuZ291Y0BnbWFpbC5jb20NCg==

------=_001_NextPart302674732371_=----
Content-Type: text/html;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

<html><head><meta http-equiv=3D"content-type" content=3D"text/html; charse=
t=3Dus-ascii"><style>body { line-height: 1.5; }body { font-size: 10.5pt; f=
ont-family: ????; color: rgb(0, 0, 0); line-height: 1.5; }</style></head><=
body><div>I can fix this by changing heap size.</div><div>But what confuse=
 me is that when i change the reducer number from 24 to 84, there's no thi=
s error.</div><div><br></div><div>Any insight on this?</div><div><br></div=
><div>Thanks</div><div>Lei</div>=0A<div><pre style=3D"font-size: 12px; lin=
e-height: normal;"><pre>Failed to merge in memoryjava.lang.OutOfMemoryErro=
r: Java heap space=0A	at java.util.Arrays.copyOf(Arrays.java:2786)=0A	at j=
ava.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)=0A	at ja=
va.io.DataOutputStream.write(DataOutputStream.java:90)=0A	at java.io.DataO=
utputStream.writeUTF(DataOutputStream.java:384)=0A	at java.io.DataOutputSt=
ream.writeUTF(DataOutputStream.java:306)=0A	at org.apache.pig.data.utils.S=
edesHelper.writeChararray(SedesHelper.java:66)=0A	at org.apache.pig.data.B=
inInterSedes.writeDatum(BinInterSedes.java:543)=0A	at org.apache.pig.data.=
BinInterSedes.writeDatum(BinInterSedes.java:435)=0A	at org.apache.pig.data=
.utils.SedesHelper.writeGenericTuple(SedesHelper.java:135)=0A	at org.apach=
e.pig.data.BinInterSedes.writeTuple(BinInterSedes.java:613)=0A	at org.apac=
he.pig.data.BinInterSedes.writeBag(BinInterSedes.java:604)=0A	at org.apach=
e.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:447)=0A	at org.apac=
he.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:435)=0A	at org.apa=
che.pig.data.utils.SedesHelper.writeGenericTuple(SedesHelper.java:135)=0A	=
at org.apache.pig.data.BinInterSedes.writeTuple(BinInterSedes.java:613)=0A=
	at org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:443)=
=0A	at org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:435=
)=0A	at org.apache.pig.data.utils.SedesHelper.writeGenericTuple(SedesHelpe=
r.java:135)=0A	at org.apache.pig.data.BinInterSedes.writeTuple(BinInterSed=
es.java:613)=0A	at org.apache.pig.data.BinInterSedes.writeDatum(BinInterSe=
des.java:443)=0A	at org.apache.pig.data.BinSedesTuple.write(BinSedesTuple.=
java:41)=0A	at org.apache.pig.impl.io.PigNullableWritable.write(PigNullabl=
eWritable.java:123)=0A	at org.apache.hadoop.io.serializer.WritableSerializ=
ation$WritableSerializer.serialize(WritableSerialization.java:100)=0A	at o=
rg.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.se=
rialize(WritableSerialization.java:84)=0A	at org.apache.hadoop.mapred.IFil=
e$Writer.append(IFile.java:188)=0A	at org.apache.hadoop.mapred.Task$Combin=
eOutputCollector.collect(Task.java:1145)=0A	at org.apache.hadoop.mapred.Ta=
sk$NewCombinerRunner$OutputConverter.write(Task.java:1456)=0A	at org.apach=
e.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputCo=
ntextImpl.java:85)=0A	at org.apache.hadoop.mapreduce.lib.reduce.WrappedRed=
ucer$Context.write(WrappedReducer.java:99)=0A	at org.apache.pig.backend.ha=
doop.executionengine.mapReduceLayer.PigCombiner$Combine.processOnePackageO=
utput(PigCombiner.java:201)=0A	at org.apache.pig.backend.hadoop.executione=
ngine.mapReduceLayer.PigCombiner$Combine.reduce(PigCombiner.java:163)=0A	a=
t org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigCombiner=
$Combine.reduce(PigCombiner.java:51)</pre></pre></div><hr style=3D"width: =
210px; height: 1px;" color=3D"#b5c4df" size=3D"1" align=3D"left">=0A<div><=
span><div style=3D"MARGIN: 10px; FONT-FAMILY: verdana; FONT-SIZE: 10pt"><d=
iv>leiwangouc@gmail.com</div></div></span></div>=0A</body></html>
------=_001_NextPart302674732371_=------