hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jiang licht <licht_ji...@yahoo.com>
Subject Re: java.io.IOException: Spill failed when using w/ GzipCodec for Map output
Date Tue, 23 Feb 2010 17:28:48 GMT
Thanks, Amogh. Good to know :)


--- On Tue, 2/23/10, Amogh Vasekar <amogh@yahoo-inc.com> wrote:

From: Amogh Vasekar <amogh@yahoo-inc.com>
Subject: Re: java.io.IOException: Spill failed when using w/ GzipCodec for Map output
To: "common-user@hadoop.apache.org" <common-user@hadoop.apache.org>
Date: Tuesday, February 23, 2010, 1:45 AM

Certainly this might not cause the issue. But,
"Hadoop native library is supported only on *nix platforms only. Unfortunately it is known
not to work on Cygwin    and Mac OS X   and has mainly been used on the  GNU/Linux platform."


The mapper log would throw more light on this


On 2/23/10 11:41 AM, "jiang licht" <licht_jiang@yahoo.com> wrote:

Thanks Amogh. The platform that I got this error is mac os x and hadoop 0.20.1. All native
library installed except lzo (which will report that codec not found). But I didn't see this
error when I ran the same thing w/o expression specified, in addition I also ran sth with
the same expression setting on Fedora 8 and 0.19.1 without any problem. So, I think it might
depends on some other settings (wrt what spill is about).



--- On Mon, 2/22/10, Amogh Vasekar <amogh@yahoo-inc.com> wrote:

From: Amogh Vasekar <amogh@yahoo-inc.com>
Subject: Re: java.io.IOException: Spill failed when using w/ GzipCodec for Map output
To: "common-user@hadoop.apache.org" <common-user@hadoop.apache.org>
Date: Monday, February 22, 2010, 11:27 PM

Can you please let us know what platform you are running on your hadoop machines?
For gzip and lzo to work, you need supported hadoop native libraries ( I remember reading
on this somewhere in hadoop wiki :) )


On 2/23/10 8:16 AM, "jiang licht" <licht_jiang@yahoo.com> wrote:

I have a pig script. If I don't set any codec for Map output for hadoop cluster, no problem.
Now I made the following compression settings, the job failed and the error message is shown
below. I guess there are some other settings that should be correctly set together with using
the compression. Im using 0.20.1. Any thoughts? Thanks for your help!


error message of failed map task--->

java.io.IOException: Spill failed
        at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:822)
        at org.apache.hadoop.mapred.MapTask$OldOutputCollector.collect(MapTask.java:466)
        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Map.collect(PigMapReduce.java:108)
        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:251)
        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:240)
        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Map.map(PigMapReduce.java:93)
        at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
        at org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by: java.lang.NullPointerException
        at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:102)
        at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1198)
        at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.access$1800(MapTask.java:648)
        at org.apache.hadoop.mapred.MapTask$MapOutputBuffer$SpillThread.run(MapTask.java:1135)



  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message