Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=g0zOSdhyWoPvTa52Tq8nU5qMoP9wk3MhGtsRUNHZ4Ey/eLADtI7LFpJgijOHXOgkCLS5x5u4Vqrfs+i2WqStuoSC6wQD/Lx0Tu4iH4npfPR88dKMfVt2sCYRRMAN18AMAwDSVB8YAqetA0TZdOHtAWis1M5meWY05TYlUTZmqBg=;
Message-ID: <825009.69756.qm@web45107.mail.sp1.yahoo.com>
Date: Tue, 23 Feb 2010 09:28:48 -0800 (PST)
From: jiang licht <licht_jiang@yahoo.com>
Subject: Re: java.io.IOException: Spill failed when using w/ GzipCodec for Map
 output
To: common-user@hadoop.apache.org
In-Reply-To: <C7A98563.7CA2%amogh@yahoo-inc.com>
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-24741262-1266946128=:69756"

--0-24741262-1266946128=:69756
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable

Thanks, Amogh. Good to know :)

=0AMichael

--- On Tue, 2/23/10, Amogh Vasekar <amogh@yahoo-inc.com> wrote:

From: Amogh Vasekar <amogh@yahoo-inc.com>
Subject: Re: java.io.IOException: Spill failed when using w/ GzipCodec for =
Map output
To: "common-user@hadoop.apache.org" <common-user@hadoop.apache.org>
Date: Tuesday, February 23, 2010, 1:45 AM

Hi,
Certainly this might not cause the issue. But,
"Hadoop native library is supported only on *nix platforms only. Unfortunat=
ely it is known not to work on Cygwin=A0 =A0 and Mac OS X=A0=A0=A0and has m=
ainly been used on the=A0 GNU/Linux platform."

http://hadoop.apache.org/common/docs/current/native_libraries.html#Supporte=
d+Platforms

The mapper log would throw more light on this

Amogh


On 2/23/10 11:41 AM, "jiang licht" <licht_jiang@yahoo.com> wrote:

Thanks Amogh. The platform that I got this error is mac os x and hadoop 0.2=
0.1. All native library installed except lzo (which will report that codec =
not found). But I didn't see this error when I ran the same thing w/o expre=
ssion specified, in addition I also ran sth with the same expression settin=
g on Fedora 8 and 0.19.1 without any problem. So, I think it might depends =
on some other settings (wrt what spill is about).

Thanks,

Michael

--- On Mon, 2/22/10, Amogh Vasekar <amogh@yahoo-inc.com> wrote:

From: Amogh Vasekar <amogh@yahoo-inc.com>
Subject: Re: java.io.IOException: Spill failed when using w/ GzipCodec for =
Map output
To: "common-user@hadoop.apache.org" <common-user@hadoop.apache.org>
Date: Monday, February 22, 2010, 11:27 PM

Hi,
Can you please let us know what platform you are running on your hadoop mac=
hines?
For gzip and lzo to work, you need supported hadoop native libraries ( I re=
member reading on this somewhere in hadoop wiki :) )

Amogh


On 2/23/10 8:16 AM, "jiang licht" <licht_jiang@yahoo.com> wrote:

I have a pig script. If I don't set any codec for Map output for hadoop clu=
ster, no problem. Now I made the following compression settings, the job fa=
iled and the error message is shown below. I guess there are some other set=
tings that should be correctly set together with using the compression. Im =
using 0.20.1. Any thoughts? Thanks for your help!

mapred-site.xml
=A0 =A0 =A0 =A0 <property>
=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 <name>mapred.compress.map.output</name>
=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 <value>true</value>
=A0 =A0 =A0 =A0 </property>
=A0 =A0 =A0 =A0 <property>
=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 <name>mapred.map.output.compression.codec</=
name>
=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 <value>org.apache.hadoop.io.compress.GzipCo=
dec</value>
=A0 =A0 =A0 =A0 </property>

error message of failed map task--->

java.io.IOException: Spill failed
=A0 =A0 =A0 =A0 at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect=
(MapTask.java:822)
=A0 =A0 =A0 =A0 at org.apache.hadoop.mapred.MapTask$OldOutputCollector.coll=
ect(MapTask.java:466)
=A0 =A0 =A0 =A0 at org.apache.pig.backend.hadoop.executionengine.mapReduceL=
ayer.PigMapReduce$Map.collect(PigMapReduce.java:108)
=A0 =A0 =A0 =A0 at org.apache.pig.backend.hadoop.executionengine.mapReduceL=
ayer.PigMapBase.runPipeline(PigMapBase.java:251)
=A0 =A0 =A0 =A0 at org.apache.pig.backend.hadoop.executionengine.mapReduceL=
ayer.PigMapBase.map(PigMapBase.java:240)
=A0 =A0 =A0 =A0 at org.apache.pig.backend.hadoop.executionengine.mapReduceL=
ayer.PigMapReduce$Map.map(PigMapReduce.java:93)
=A0 =A0 =A0 =A0 at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50=
)
=A0 =A0 =A0 =A0 at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.ja=
va:358)
=A0 =A0 =A0 =A0 at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
=A0 =A0 =A0 =A0 at org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by: java.lang.NullPointerException
=A0 =A0 =A0 =A0 at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:102)
=A0 =A0 =A0 =A0 at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAnd=
Spill(MapTask.java:1198)
=A0 =A0 =A0 =A0 at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.access$=
1800(MapTask.java:648)
=A0 =A0 =A0 =A0 at org.apache.hadoop.mapred.MapTask$MapOutputBuffer$SpillTh=
read.run(MapTask.java:1135)


Thanks,

Michael


=0A=0A=0A      
--0-24741262-1266946128=:69756--