From user-return-2703-apmail-hadoop-user-archive=hadoop.apache.org@hadoop.apache.org Wed Nov 7 01:54:46 2012 Return-Path: X-Original-To: apmail-hadoop-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 15715D7C2 for ; Wed, 7 Nov 2012 01:54:46 +0000 (UTC) Received: (qmail 53878 invoked by uid 500); 7 Nov 2012 01:54:41 -0000 Delivered-To: apmail-hadoop-user-archive@hadoop.apache.org Received: (qmail 53754 invoked by uid 500); 7 Nov 2012 01:54:41 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 53744 invoked by uid 99); 7 Nov 2012 01:54:41 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Nov 2012 01:54:41 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of arpit@hortonworks.com designates 209.85.220.48 as permitted sender) Received: from [209.85.220.48] (HELO mail-pa0-f48.google.com) (209.85.220.48) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Nov 2012 01:54:36 +0000 Received: by mail-pa0-f48.google.com with SMTP id kp12so786531pab.35 for ; Tue, 06 Nov 2012 17:54:16 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=from:content-type:message-id:mime-version:subject:date:references :to:in-reply-to:x-mailer:x-gm-message-state; bh=IlLfSHj/QzNhMnz6pwiBe3tmvb+2Tgtp8o7AK91muXU=; b=BD4whDlzhcGAodDcJC32gGb/JZx1xnPUmjwmKuYna+X/8TcCcSGTZ4vJmVbJAeGHS/ DTud4q7m1Mo0HHMFIgQh8LOe6nDPZQR90Acsc7AaJjWTumJfptgNbAkaFC+e9XtNZ9Ct t7Kj25ozT5yxEa1c18TCcItg9CrLHdLYrWzNk/ILEq3TL0lj6U0yBXDR3s43AUgtnZ+U qgfNZa48/+IAK9xF06ULcqHebnpAuqADxyAAyi1vW52TdNvfyyb0zzDk7/bEKmhOQHXI u2MVqwG2KZjlyPf6mwdAJ3SSYU13eFz/qyrM7pwPiKHYPEwSwE9OyZDI38PfPNHmmQeY 7fTw== Received: by 10.68.226.136 with SMTP id rs8mr2438773pbc.153.1352253255312; Tue, 06 Nov 2012 17:54:15 -0800 (PST) Received: from [192.168.1.105] (c-71-202-190-17.hsd1.ca.comcast.net. [71.202.190.17]) by mx.google.com with ESMTPS id qi2sm13224416pbc.22.2012.11.06.17.54.13 (version=TLSv1/SSLv3 cipher=OTHER); Tue, 06 Nov 2012 17:54:14 -0800 (PST) From: Arpit Gupta Content-Type: multipart/alternative; boundary="Apple-Mail=_B478342F-0418-4186-BE35-790CC01021FB" Message-Id: Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\)) Subject: Re: problem with hadoop-snappy Date: Tue, 6 Nov 2012 17:55:18 -0800 References: <8CF89D299ECF297-1F14-14CAB@webmail-m164.sysops.aol.com> <8CF8A7D17105E5D-1F14-1C777@webmail-m164.sysops.aol.com> <8CF8AA0538297BA-1F14-1DA6E@webmail-m164.sysops.aol.com> To: user@hadoop.apache.org In-Reply-To: <8CF8AA0538297BA-1F14-1DA6E@webmail-m164.sysops.aol.com> X-Mailer: Apple Mail (2.1499) X-Gm-Message-State: ALoCoQn4obVgzE16WU8OIf87i9NCRHukPZdRdHj48bRj6wt33ssEx24cs1eSJiiRn758EfQ2bZPs X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail=_B478342F-0418-4186-BE35-790CC01021FB Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Alex You need to make sure libsnappy.so is available in hadoop's java library = path. The way hadoop sets the java library path is that based on the jdk it = will add either 32bit or 64bit libraries. They need to be present in $HADOOP_HOME/lib/native/Linux-amd64-64 for = 64bit jdk or $HADOOP_HOME/lib/native/Linux-i386-32 for 32 bit jdk. So = create the libsnappy.so link in the appropriate directories on all your = nodes restart your tasktrackers and submit your job again. -- Arpit Gupta Hortonworks Inc. http://hortonworks.com/ On Nov 6, 2012, at 5:42 PM, alxsss@aim.com wrote: > Yes, I have installed snappy-1.0.5. >=20 > Here is the output of=20 > ls -l /usr/local/lib > total 1468 > lrwxrwxrwx. 1 root root 52 Nov 5 16:00 libjvm.so -> = /usr/java/jdk1.6.0_37/jre/lib/amd64/server/libjvm.so > -rw-r--r--. 1 root root 946358 Nov 3 17:44 liblzo2.a > -rwxr-xr-x. 1 root root 866 Nov 3 17:44 liblzo2.la > -rw-r--r--. 1 root root 371174 Nov 5 15:18 libsnappy.a > -rwxr-xr-x. 1 root root 957 Nov 5 15:18 libsnappy.la > lrwxrwxrwx. 1 root root 18 Nov 5 15:18 libsnappy.so -> = libsnappy.so.1.1.3 > lrwxrwxrwx. 1 root root 18 Nov 5 15:18 libsnappy.so.1 -> = libsnappy.so.1.1.3 > -rwxr-xr-x. 1 root root 169393 Nov 5 15:18 libsnappy.so.1.1.3 >=20 >=20 >=20 > -----Original Message----- > From: Binglin Chang > To: user > Sent: Tue, Nov 6, 2012 5:34 pm > Subject: Re: problem with hadoop-snappy >=20 > Have you installed snappy library (which should have libsnappy.so in = /usr/lib or /usr/local/lib) first? >=20 >=20 >=20 > On Wed, Nov 7, 2012 at 5:30 AM, wrote: > When I used hadoop-1.0.4 without the installation of hadoop-snappy I = saw in the logs >=20 > WARN snappy.LoadSnappy: Snappy native library not loaded >=20 > So, it was my understanding that snappy is not included with = hadoop-1.0.4. After removing the libs I installed, I see again the same = warning. >=20 > Any ideas how to fix this issue? >=20 > Thanks. > Alex. >=20 >=20 >=20 >=20 > -----Original Message----- > From: Binglin Chang > To: user > Sent: Mon, Nov 5, 2012 6:07 pm > Subject: Re: problem with hadoop-snappy >=20 > I think hadoop-1.0.4 already have snappy included, you should not = using other third party libraries. > =20 >=20 >=20 > On Tue, Nov 6, 2012 at 9:10 AM, wrote: > Hello, >=20 > I use hadoop-1.0.4 I have followed instruction to install = hadoop-snappy at=20 > http://code.google.com/p/hadoop-snappy/=20 >=20 > When I run a mapred job I see >=20 > FATAL org.apache.hadoop.mapred.TaskTracker: Task: = attempt_201211051656_0002_m_000000_3 - Killed : = org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressBytesDirect(= )I >=20 > in task tracker log and the job fails. >=20 > Any ideas how to fix this issue. >=20 > Thanks in advance. > Alex. >=20 >=20 >=20 --Apple-Mail=_B478342F-0418-4186-BE35-790CC01021FB Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii
--
Arpit Gupta
Hortonworks Inc.
http://hortonworks.com/
<= /span>

On Nov 6, 2012, at 5:42 PM, alxsss@aim.com wrote:

Yes, I have installed snappy-1.0.5.

Here is the output of
 ls -l /usr/local/lib
total 1468
lrwxrwxrwx. 1 root root     52 Nov  5 16:00 = libjvm.so -> /usr/java/jdk1.6.0_37/jre/lib/amd64/server/libjvm.so
-rw-r--r--. 1 root root 946358 Nov  3 17:44 liblzo2.a
-rwxr-xr-x. 1 root root    866 Nov  3 17:44 = liblzo2.la
-rw-r--r--. 1 root root 371174 Nov  5 15:18 libsnappy.a
-rwxr-xr-x. 1 root root    957 Nov  5 15:18 = libsnappy.la
lrwxrwxrwx. 1 root root     18 Nov  5 15:18 = libsnappy.so -> libsnappy.so.1.1.3
lrwxrwxrwx. 1 root root     18 Nov  5 15:18 = libsnappy.so.1 -> libsnappy.so.1.1.3
-rwxr-xr-x. 1 root root 169393 Nov  5 15:18 libsnappy.so.1.1.3



-----Original Message-----
From: Binglin Chang <decstery@gmail.com>
To: user <user@hadoop.apache.org>
Sent: Tue, Nov 6, 2012 5:34 pm
Subject: Re: problem with hadoop-snappy

Have you installed snappy library (which should have libsnappy.so in = /usr/lib or /usr/local/lib) first?



On Wed, Nov = 7, 2012 at 5:30 AM, <alxsss@aim.com> wrote:
When I used = hadoop-1.0.4  without the installation of hadoop-snappy = I saw in the logs

 WARN snappy.LoadSnappy: Snappy native library not loaded

So, it was my understanding that snappy is not = included with hadoop-1.0.4. After removing the libs I installed, I see = again the same warning.

Any ideas how to fix this issue?

Thanks.
Alex.
=




-----Original Message-----
From: Binglin Chang <decstery@gmail.com>
To: user <user@hadoop.apache.org>
Sent: Mon, Nov 5, 2012 6:07 pm
Subject: Re: problem with hadoop-snappy

I think hadoop-1.0.4 already have snappy included, you should not using = other third party libraries.
 


On Tue, Nov 6, 2012 at = 9:10 AM, <alxsss@aim.com> wrote:
Hello,

I use hadoop-1.0.4  I have followed  instruction = to install hadoop-snappy at
http://code.google.com/p/= hadoop-snappy/


When I run a mapred job I see

FATAL org.apache.hadoop.mapred.TaskTracker: Task: = attempt_201211051656_0002_m_000000_3 - Killed : = org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressBytesDirect(= )I

in task tracker log and the job fails.

Any ideas how to fix this issue.

Thanks in advance.
Alex.


=20


= --Apple-Mail=_B478342F-0418-4186-BE35-790CC01021FB--