Return-Path: Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: (qmail 30593 invoked from network); 23 Feb 2010 05:28:13 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 23 Feb 2010 05:28:13 -0000 Received: (qmail 88557 invoked by uid 500); 23 Feb 2010 05:28:10 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 88403 invoked by uid 500); 23 Feb 2010 05:28:09 -0000 Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: common-user@hadoop.apache.org Delivered-To: mailing list common-user@hadoop.apache.org Received: (qmail 88393 invoked by uid 99); 23 Feb 2010 05:28:09 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 23 Feb 2010 05:28:09 +0000 X-ASF-Spam-Status: No, hits=4.2 required=10.0 tests=HTML_MESSAGE,NO_RDNS_DOTCOM_HELO,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: local policy) Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 23 Feb 2010 05:27:58 +0000 Received: from EGL-EX07CAS01.ds.corp.yahoo.com (egl-ex07cas01.eglbp.corp.yahoo.com [203.83.248.208]) by mrout1-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o1N5R6Bd063050 for ; Mon, 22 Feb 2010 21:27:07 -0800 (PST) DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns; h=received:from:to:date:subject:thread-topic:thread-index: message-id:in-reply-to:accept-language:content-language: x-ms-has-attach:x-ms-tnef-correlator:acceptlanguage:content-type:mime-version; b=sywVWWKFPZ9Rxuzmc6fFJbl7C75f186LMpiqrUecgBO/iF3jjFvpf6DJb1Uy4oUo Received: from EGL-EX07VS01.ds.corp.yahoo.com ([203.83.248.205]) by EGL-EX07CAS01.ds.corp.yahoo.com ([203.83.248.215]) with mapi; Tue, 23 Feb 2010 10:57:06 +0530 From: Amogh Vasekar To: "common-user@hadoop.apache.org" Date: Tue, 23 Feb 2010 10:57:04 +0530 Subject: Re: java.io.IOException: Spill failed when using w/ GzipCodec for Map output Thread-Topic: java.io.IOException: Spill failed when using w/ GzipCodec for Map output Thread-Index: Acq0MpF+PQocdwG1QzGwXAY4388iuAAFkOkG Message-ID: In-Reply-To: <824868.77334.qm@web45116.mail.sp1.yahoo.com> Accept-Language: en-US Content-Language: en X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: multipart/alternative; boundary="_000_C7A965007C87amoghyahooinccom_" MIME-Version: 1.0 X-Virus-Checked: Checked by ClamAV on apache.org --_000_C7A965007C87amoghyahooinccom_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Hi, Can you please let us know what platform you are running on your hadoop mac= hines? For gzip and lzo to work, you need supported hadoop native libraries ( I re= member reading on this somewhere in hadoop wiki :) ) Amogh On 2/23/10 8:16 AM, "jiang licht" wrote: I have a pig script. If I don't set any codec for Map output for hadoop clu= ster, no problem. Now I made the following compression settings, the job fa= iled and the error message is shown below. I guess there are some other set= tings that should be correctly set together with using the compression. Im = using 0.20.1. Any thoughts? Thanks for your help! mapred-site.xml mapred.compress.map.output true mapred.map.output.compression.codec org.apache.hadoop.io.compress.GzipCodec error message of failed map task---> java.io.IOException: Spill failed at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask= .java:822) at org.apache.hadoop.mapred.MapTask$OldOutputCollector.collect(MapT= ask.java:466) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Pig= MapReduce$Map.collect(PigMapReduce.java:108) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Pig= MapBase.runPipeline(PigMapBase.java:251) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Pig= MapBase.map(PigMapBase.java:240) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Pig= MapReduce$Map.map(PigMapReduce.java:93) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307) at org.apache.hadoop.mapred.Child.main(Child.java:170) Caused by: java.lang.NullPointerException at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:102) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(Ma= pTask.java:1198) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.access$1800(Map= Task.java:648) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer$SpillThread.run= (MapTask.java:1135) Thanks, Michael --_000_C7A965007C87amoghyahooinccom_--