Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=DaCfclCmZYBIGQndrA9JzlEdpi7g92MTOuiYexQvrdtShNVcYcwOoteWf5fsn3a56F+a7WHwqUC5Uxq3zBMyvVJ9ZIFeKwTEAkJ6yI96R9kjxqN5FV/3eOahwb+06YAoIL3zgVyP4nqbYLZFGVB80zM+TBlxIAamVnIZDBznXMQ=;
Message-ID: <257246.2073.qm@web45108.mail.sp1.yahoo.com>
Date: Mon, 22 Feb 2010 18:36:31 -0800 (PST)
From: jiang licht <licht_jiang@yahoo.com>
Subject: Re: OutOfMemoryError of PIG job (UDF loads big file)
To: common-user@hadoop.apache.org
In-Reply-To: <8211a1321002221813g6d2be728wfd33058e0b623ee3@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-219683985-1266892591=:2073"

--0-219683985-1266892591=:2073
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable

Thanks Jeff. I also just found this one and solved my problem. BTW, so many=
 settings to play with :)

=0AMichael

--- On Mon, 2/22/10, Jeff Zhang <zjffdu@gmail.com> wrote:

From: Jeff Zhang <zjffdu@gmail.com>
Subject: Re: OutOfMemoryError of PIG job (UDF loads big file)
To: common-user@hadoop.apache.org
Date: Monday, February 22, 2010, 8:13 PM

Hi Jiang,

you should set property *mapred.child.java.opts* in mapred-site.xml to
increase the memeory
as following:

 <property>
=A0 =A0 =A0 =A0=A0=A0<name>mapred.child.java.opts</name>
=A0 =A0 =A0 =A0=A0=A0<value>-Xmx1024m</value>
 </property>

 and then restart your hadoop cluster


On Tue, Feb 23, 2010 at 9:43 AM, jiang licht <licht_jiang@yahoo.com> wrote:

> I am running a hadoop job written in PIG. It fails from out of memory
> because a UDF function consumes a lot of memory, it loads a big file. Wha=
t
> are the settings to avoid the following OutOfMemoryError? I guess by simp=
ly
> giving PIG big memory (java -XmxBIGmemory org.apache.pig.Main ...) won't
> work.
>
> Error message --->
>
> java.lang.OutOfMemoryError: Java heap space
>=A0 =A0 =A0 =A0 at java.util.regex.Pattern.compile(Pattern.java:1451)
>=A0 =A0 =A0 =A0 at java.util.regex.Pattern.(Pattern.java:1133)
>=A0 =A0 =A0 =A0 at java.util.regex.Pattern.compile(Pattern.java:823)
>=A0 =A0 =A0 =A0 at java.lang.String.split(String.java:2293)
>=A0 =A0 =A0 =A0 at java.lang.String.split(String.java:2335)
>=A0 =A0 =A0 =A0 at UDF.load(Unknown Source)
>=A0 =A0 =A0 =A0 at UDF.load(Unknown Source)
>=A0 =A0 =A0 =A0 at UDF.exec(Unknown Source)
>=A0 =A0 =A0 =A0 at UDF.exec(Unknown Source)
>=A0 =A0 =A0 =A0 at
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOpe=
rators.POUserFunc.getNext(POUserFunc.java:201)
>=A0 =A0 =A0 =A0 at
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOpe=
rators.POUserFunc.getNext(POUserFunc.java:287)
>=A0 =A0 =A0 =A0 at
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOpe=
rators.POForEach.processPlan(POForEach.java:278)
>=A0 =A0 =A0 =A0 at
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOpe=
rators.POForEach.getNext(POForEach.java:204)
>=A0 =A0 =A0 =A0 at
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOpera=
tor.processInput(PhysicalOperator.java:231)
>=A0 =A0 =A0 =A0 at
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOpe=
rators.POLocalRearrange.getNext(POLocalRearrange.java:240)
>=A0 =A0 =A0 =A0 at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.r=
unPipeline(PigMapBase.java:249)
>=A0 =A0 =A0 =A0 at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.m=
ap(PigMapBase.java:240)
>=A0 =A0 =A0 =A0 at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce=
$Map.map(PigMapReduce.java:93)
>=A0 =A0 =A0 =A0 at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:5=
0)
>=A0 =A0 =A0 =A0 at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
>=A0 =A0 =A0 =A0 at org.apache.hadoop.mapred.Child.main(Child.java:155)
>
> Thanks!
> Michael
>
>
>
>


--=20
Best Regards

Jeff Zhang
=0A=0A=0A      
--0-219683985-1266892591=:2073--