Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 03F2E10D09 for ; Fri, 4 Oct 2013 19:45:40 +0000 (UTC) Received: (qmail 5647 invoked by uid 500); 4 Oct 2013 19:45:17 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 5121 invoked by uid 500); 4 Oct 2013 19:45:05 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 5112 invoked by uid 99); 4 Oct 2013 19:45:04 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Oct 2013 19:45:04 +0000 X-ASF-Spam-Status: No, hits=2.4 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of java8964@hotmail.com designates 65.55.90.94 as permitted sender) Received: from [65.55.90.94] (HELO snt0-omc2-s19.snt0.hotmail.com) (65.55.90.94) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Oct 2013 19:44:56 +0000 Received: from SNT149-W19 ([65.55.90.73]) by snt0-omc2-s19.snt0.hotmail.com with Microsoft SMTPSVC(6.0.3790.4675); Fri, 4 Oct 2013 12:44:35 -0700 X-TMN: [YFv7d+gjQ87lsrPwybvU0knKOjaUvE5aWnBADwrRAS0=] X-Originating-Email: [java8964@hotmail.com] Message-ID: Content-Type: multipart/alternative; boundary="_116a21ba-9215-40d6-90d4-e260b5c1c271_" From: java8964 java8964 To: "user@hadoop.apache.org" Subject: enable snappy on hadoop 1.1.1 Date: Fri, 4 Oct 2013 15:44:34 -0400 Importance: Normal MIME-Version: 1.0 X-OriginalArrivalTime: 04 Oct 2013 19:44:35.0148 (UTC) FILETIME=[2745E8C0:01CEC13A] X-Virus-Checked: Checked by ClamAV on apache.org --_116a21ba-9215-40d6-90d4-e260b5c1c271_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Hi=2C=20 I am using hadoop 1.1.1. I want to test to see the snappy compression with = hadoop=2C but I have some problems to make it work on my Linux environment. I am using opensuse 12.3 x86_64.=20 First=2C when I tried to enable snappy in hadoop 1.1.1 by: conf.setBoolean("mapred.compress.map.output"=2C true)=3B = conf.set("mapred.output.compression.type"=2C "RECORD")=3B c= onf.set("mapred.map.output.compression.codec"=2C "org.apache.hadoop.io.comp= ress.SnappyCodec")=3B I got the following error in my test MR job: Exception in thread "main" java.lang.RuntimeException: native snappy librar= y not available So I download the snappy 1.1.0 from https://code.google.com/p/snappy/=2C co= mpile it and install it successfully under /opt/snappy-1.1.0=2C and then I = link the /opt/snappy-1.1.0/lib64/libsnappy.so to /user/lib64/libsnappy.so Now after I restart the hadoop and tried my test MR job again=2C this time= =2C it didn't give me the originally error=2C but a new error like this: Error: java.lang.UnsatisfiedLinkError: org/apache/hadoop/io/compress/snappy= /SnappyCompressor.compressBytesDirect()I at org.apache.hadoop.io.compress.s= nappy.SnappyCompressor.compress(SnappyCompressor.java:229) at org.apache.ha= doop.io.compress.BlockCompressorStream.compress(BlockCompressorStream.java:= 141) at org.apache.hadoop.io.compress.BlockCompressorStream.finish(BlockCom= pressorStream.java:135) at org.apache.hadoop.mapred.IFile$Writer.close(IFil= e.java:135) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpil= l(MapTask.java:1450) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.ac= cess$1800(MapTask.java:852) at org.apache.hadoop.mapred.MapTask$MapOutputBu= ffer$SpillThread.run(MapTask.java:1343) I wrote a test problem=2C like hadoop did load the library: System.loadlibrary("snappy") it works fine in my test program. I don't know why at runtime=2C the Class SnappyCompressor.compressByteDirec= t() gave back that kind of error. From the source code=2C it looks like a n= ative c program from here https://code.google.com/p/hadoop-snappy/source/browse/trunk/src/main/java/o= rg/apache/hadoop/io/compress/snappy/SnappyCompressor.java Any idea why this is happening on my environment? Thanks Yong = --_116a21ba-9215-40d6-90d4-e260b5c1c271_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable
Hi=2C =3B

I am using hadoop 1.1.1. I want to test to see the snappy compression with= hadoop=2C but I have some problems to make it work on my Linux environment= .

I am using opensuse 12.3 x86_64. =3B

First=2C when I tried to enable snappy in hadoop 1.1.1 by= :

 =3B  =3B  =3B  =3B  = =3B  =3B conf.setBoolean("mapred.compress.map.output"=2C true)=3B
=
 =3B  =3B  =3B  =3B  =3B  =3B conf.set("mapred= .output.compression.type"=2C "RECORD")=3B
 =3B  =3B  = =3B  =3B  =3B  =3B conf.set("mapred.map.output.compression.code= c"=2C "org.apache.hadoop.io.compress.SnappyCodec")=3B

<= /div>
I got the following error in my test MR job:

=
Exception in thread "main" java.lang.RuntimeException: native snappy l= ibrary not available

So I download the snappy 1.1.= 0 from =3Bhttps://code.google.com/p/snappy/=2C co= mpile it and install it successfully under /opt/snappy-1.1.0=2C and then I = link the /opt/snappy-1.1.0/lib64/libsnappy.so to /user/lib64/libsnappy.so

Now after I restart the hadoop and tried my test MR= job again=2C this time=2C it didn't give me the originally error=2C but a = new error like this:

Error: java.lang.Unsatis= fiedLinkError: org/apache/hadoop/io/compress/snappy/SnappyCompressor.compre= ssBytesDirect()I
at org.apache.hadoop.io.compress.snappy.SnappyCompress= or.compress(SnappyCompressor.java:229)
at org.apache.hadoop.io.compress= .BlockCompressorStream.compress(BlockCompressorStream.java:141)
<= span class=3D"Apple-tab-span" style=3D"white-space: pre=3B"> at org.= apache.hadoop.io.compress.BlockCompressorStream.finish(BlockCompressorStrea= m.java:135)
at org.apache.hadoop.mapred.IFile$Writer.close(IFile.java:1= 35)
= at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(Ma= pTask.java:1450)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.ac= cess$1800(MapTask.java:852)
at org.apache.hadoop.mapred.MapTask$MapOut= putBuffer$SpillThread.run(MapTask.java:1343)

I wrote a test problem=2C like hadoop did load the library:

=
System.loadlibrary("snappy")

it works f= ine in my test program.

I don't know why at runtim= e=2C the Class SnappyCompressor.compressByteDirect() gave back that kind of= error. From the source code=2C it looks like a native c program from here<= /div>


Any idea why this is happ= ening on my environment?

Thanks

Yong
= --_116a21ba-9215-40d6-90d4-e260b5c1c271_--