Return-Path: X-Original-To: apmail-hadoop-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id AB7C61093D for ; Thu, 2 Jan 2014 11:58:23 +0000 (UTC) Received: (qmail 72204 invoked by uid 500); 2 Jan 2014 11:57:34 -0000 Delivered-To: apmail-hadoop-user-archive@hadoop.apache.org Received: (qmail 72037 invoked by uid 500); 2 Jan 2014 11:57:13 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 71972 invoked by uid 99); 2 Jan 2014 11:57:10 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jan 2014 11:57:10 +0000 X-ASF-Spam-Status: No, hits=1.7 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of bharathvissapragada1990@gmail.com designates 209.85.128.53 as permitted sender) Received: from [209.85.128.53] (HELO mail-qe0-f53.google.com) (209.85.128.53) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jan 2014 11:57:04 +0000 Received: by mail-qe0-f53.google.com with SMTP id nc12so14261637qeb.12 for ; Thu, 02 Jan 2014 03:56:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type; bh=xFgOY33W8xDTUaC7nsuDqZeuSRsrtJpapU/ub8K6M3I=; b=u+ExPRgqh5B1QoZr2fEKiOXg+8Jco9sTU4OfDjNixbsbmOisfcXCoQ14PL4jWy3zla msT7DvQcK3MQT+WvYuV11vQw5p0z0wSnkJlUXeeXrTMqWhfSnUZpRsntVznZam7OqKRS zTgwnEeRZpDkegDofFoZecnHoNSurt3E1X3+Cage1cTdo9lCy1CRObhESKg3ejAqp7W2 f4t4J1snA68eK7gvqZ2F4dItGk0yy9TRQbEZry8MhEq80eTYUZL3/zKjxzUx0AE8nIXZ jcHa1Ztp6YzOxqwlv/RlrLMDehKACZPaZgNAOHh9fHmNkfzUJWF/gVWRsb4EtsEhvrPm pctg== X-Received: by 10.49.12.43 with SMTP id v11mr140722298qeb.50.1388663803789; Thu, 02 Jan 2014 03:56:43 -0800 (PST) MIME-Version: 1.0 Received: by 10.140.95.109 with HTTP; Thu, 2 Jan 2014 03:56:23 -0800 (PST) In-Reply-To: References: From: bharath vissapragada Date: Thu, 2 Jan 2014 17:26:23 +0530 Message-ID: Subject: Re: Setting up Snappy compression in Hadoop To: User Content-Type: multipart/alternative; boundary=047d7b6d8d14bcc58404eefb7c44 X-Virus-Checked: Checked by ClamAV on apache.org --047d7b6d8d14bcc58404eefb7c44 Content-Type: text/plain; charset=ISO-8859-1 Your natives should be in LD_LIBRARY_PATH or java.library.path for hadoop to pick them up. You can try adding export HADOOP_OPTS=$HADOOP_OPTS -Djava.library.path= to hadoop-env.sh in TTs and clients/gateways and restart TTs and give it another try. The reason its working for Hbase is you are manually pointing HBASE_LIBRARY_PATH to the natives. My guess is they are in a wrong location. On Thu, Jan 2, 2014 at 5:07 PM, Amit Sela wrote: > I did everything mentioned in the link Ted mentioned, and the test > actually works, but using Snappy for MapReduce map output compression still > fails with "native snappy library not available". > > > On Wed, Jan 1, 2014 at 6:37 PM, bharath vissapragada < > bharathvissapragada1990@gmail.com> wrote: > >> Did you build it for your platform? You can do an "ldd" on the .so file >> to check if the dependent libs are present. Also make sure you placed it in >> the right directory for your platform (Linux-amd64-64 or Linux-i386-32) >> >> >> On Wed, Jan 1, 2014 at 10:02 PM, Ted Yu wrote: >> >>> Please take a look at >>> http://hbase.apache.org/book.html#snappy.compression >>> >>> Cheers >>> >>> >>> On Wed, Jan 1, 2014 at 8:05 AM, Amit Sela wrote: >>> >>>> Hi all, >>>> >>>> I'm running on Hadoop 1.0.4 and I'd like to use Snappy for map output >>>> compression. >>>> I'm adding the configurations: >>>> >>>> configuration.setBoolean("mapred.compress.map.output", true); >>>> configuration.set("mapred.map.output.compression.codec", >>>> "org.apache.hadoop.io.compress.SnappyCodec"); >>>> >>>> And I've added libsnappy.so.1 to $HADOOP_HOME/lib/native/Linux-amd64-64/ >>>> >>>> Still, all map tasks fail with "native snappy library not available". >>>> >>>> Could anyone elaborate on how to install Snappy for Hadoop ? >>>> >>>> Thanks, >>>> >>>> Amit. >>>> >>> >>> >> > --047d7b6d8d14bcc58404eefb7c44 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Your natives should be in LD_LIBRARY_PATH or=A0java.librar= y.path for hadoop to pick them up. You can try adding export HADOOP_OPTS=3D= $HADOOP_OPTS -Djava.library.path=3D<Path to your natives lib> to hado= op-env.sh in TTs and clients/gateways and restart TTs and give it another t= ry. The reason its working for Hbase is you are manually pointing=A0HBASE_L= IBRARY_PATH to=A0the natives. My guess is they are in a wrong location.


On Thu, Jan 2= , 2014 at 5:07 PM, Amit Sela <amits@infolinks.com> wrote:<= br>
I did everything mentioned in the link Ted mentioned, and = the test actually works, but using Snappy for MapReduce map output compress= ion still fails with "native snappy library not available".


On Wed, Jan 1= , 2014 at 6:37 PM, bharath vissapragada <bharathvissapraga= da1990@gmail.com> wrote:
Did you build it for your p= latform? You can do an "ldd" on the .so file to check if the depe= ndent libs are present. Also make sure you placed it in the right directory= for your platform (Linux-amd64-64 or Linux-i386-32)


On Wed, Jan 1= , 2014 at 10:02 PM, Ted Yu <yuzhihong@gmail.com> wrote:


On Wed, Jan 1, 2014 at 8:05 AM, Amit Sela <amits@infolinks.com> wrote:
Hi all,=A0

I'm running on Hadoop 1.= 0.4 and I'd like to use Snappy for map output compression.
I&= #39;m adding the configurations:

configuration.set= Boolean("mapred.compress.map.output", true);
configuration.set("mapred.map.output.compression.codec"= ;, "org.apache.hadoop.io.compress.SnappyCodec");
And I've added=A0libsnappy.so.1 to $HADOOP_HOME/lib/native= /Linux-amd64-64/

Still, all map tasks fail with "native snapp= y library not available".

Could anyone elaborate on how to install Snappy for Hadoop ?<= /div>

Thanks,=A0

Amit.
<= /font>




--047d7b6d8d14bcc58404eefb7c44--