hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Marc Spaggiari <jean-m...@spaggiari.org>
Subject Re: How to install Snappy?
Date Mon, 03 Dec 2012 14:56:46 GMT
Hi Kevin,

Thanks for the clarification.

No, it's not what I'm seeing.

Here is what I'm getting:

12/12/03 09:40:42 WARN snappy.LoadSnappy: Snappy native library is available
12/12/03 09:40:42 WARN snappy.LoadSnappy: Snappy native library not loaded
Exception in thread "main" java.lang.RuntimeException: native snappy
library not available
	at org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:123)
	at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:100)
	at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:112)
	at org.apache.hadoop.hbase.io.hfile.Compression$Algorithm.getCompressor(Compression.java:264)
	at org.apache.hadoop.hbase.io.hfile.HFileBlock$Writer.<init>(HFileBlock.java:739)
	at org.apache.hadoop.hbase.io.hfile.HFileWriterV2.finishInit(HFileWriterV2.java:127)
	at org.apache.hadoop.hbase.io.hfile.HFileWriterV2.<init>(HFileWriterV2.java:118)
	at org.apache.hadoop.hbase.io.hfile.HFileWriterV2$WriterFactoryV2.createWriter(HFileWriterV2.java:101)
	at org.apache.hadoop.hbase.io.hfile.HFile$WriterFactory.create(HFile.java:394)
	at org.apache.hadoop.hbase.util.CompressionTest.doSmokeTest(CompressionTest.java:108)
	at org.apache.hadoop.hbase.util.CompressionTest.main(CompressionTest.java:138)

The most disturbing part is this line:
12/12/03 09:40:42 WARN snappy.LoadSnappy: Snappy native library is available

Followed by this one:
Exception in thread "main" java.lang.RuntimeException: native snappy
library not available

Is it available? Or is it not available?

I looked in the code and I have no idea why the 2nd one is raised.

The code I'm looking at is on the hadoop-snappy site, but the one I
have on my server is in the hadoop the hadoop-core-1.0.3.jar file. So
maybe that the issue and they are different?

I built the hadoop-snappy-0.0.1-SNAPSHOT.jar file too. I placed it on
the lib folder and made sure it was taken first, but still not
working.

So far I think I will stay with GZip until Snappy is integrated on the
HBase files...

JM

2012/12/3, Kevin O'dell <kevin.odell@cloudera.com>:
> Hey JM,
>
>   Sorry for the quick message earlier.  I tracked down the JIRA I was
> referring to: https://issues.apache.org/jira/browse/HBASE-7080
>
> Does this look like what you are seeing in Compression test?
>
> On Mon, Dec 3, 2012 at 9:09 AM, Kevin O'dell
> <kevin.odell@cloudera.com>wrote:
>
>> There is a compression test JIRA right now.  What are you seeing?
>>
>>
>> On Mon, Dec 3, 2012 at 8:47 AM, Jean-Marc Spaggiari <
>> jean-marc@spaggiari.org> wrote:
>>
>>> Ok....
>>>
>>> This: http://code.google.com/p/hadoop-snappy/issues/detail?id=2 helped
>>> me and my test program is now working. I'm able to load both
>>> libraries. Fine.
>>>
>>> But the CompressionTest is still not working.
>>>
>>> What is very strange is that:
>>> 12/12/03 08:44:24 WARN snappy.LoadSnappy: Snappy native library is
>>> available
>>> 12/12/03 08:44:24 WARN snappy.LoadSnappy: Snappy native library not
>>> loaded
>>>
>>> It's available, but not loaded.
>>>
>>> But from the code:
>>>   static {
>>>     try {
>>>       System.loadLibrary("snappy");
>>>       System.loadLibrary("hadoopsnappy");
>>>       LOG.warn("Snappy native library is available");
>>>       AVAILABLE = true;
>>>     } catch (UnsatisfiedLinkError ex) {
>>>       //NOP
>>>     }
>>>     LOADED = AVAILABLE;
>>>     if (LOADED) {
>>>       LOG.info("Snappy native library loaded");
>>>     } else {
>>>       LOG.warn("Snappy native library not loaded");
>>>     }
>>>   }
>>> If "Snappy native library is available" is displayed, that mean
>>> AVAILABLE = true... And if AVAILABLE = true, then LOADED is set to
>>> true and Snappy native library loaded must be displayed... But it's
>>> not... How is this possible?
>>>
>>> I have not expected Snappy installation to be such a challenge...
>>>
>>> I will continue to dig and summarize the steps when I will be done (If
>>> I'm able to finish...)
>>>
>>> JM
>>>
>>> 2012/12/3, Jean-Marc Spaggiari <jean-marc@spaggiari.org>:
>>> > Thanks all for your replies.
>>> >
>>> > So, to reply to all in one.
>>> >
>>> > I'm not using CD3. I'm using Hadoop  1.0.3 and HBase 0.94.2 directly
>>> > from the JARs.
>>> >
>>> > Here are all the places where I have put the lib:
>>> > /home/hadoop/hadoop-1.0.3/lib/native/Linux-amd64-64/libsnappy.so
>>> > /home/hadoop/hadoop-1.0.3/lib/native/Linux-amd64-64/libsnappy.so.1
>>> > /home/hadoop/hadoop-1.0.3/lib/native/Linux-amd64-64/libsnappy.so.1.1.3
>>> > /home/hadoop/hadoop-1.0.3/lib/native/Linux-i386-32/libsnappy.so
>>> > /home/hadoop/hadoop-1.0.3/lib/native/Linux-i386-32/libsnappy.so.1
>>> > /home/hadoop/hadoop-1.0.3/lib/native/Linux-i386-32/libsnappy.so.1.1.3
>>> > /home/hbase/hbase-0.94.2/lib/native/libsnappy.so
>>> > /home/hbase/hbase-0.94.2/lib/native/libsnappy.so.1
>>> > /home/hbase/hbase-0.94.2/lib/native/libsnappy.so.1.1.3
>>> > /home/hbase/hbase-0.94.2/lib/native/Linux-amd64-64/libsnappy.so
>>> > /home/hbase/hbase-0.94.2/lib/native/Linux-amd64-64/libsnappy.so.1
>>> > /home/hbase/hbase-0.94.2/lib/native/Linux-amd64-64/libsnappy.so.1.1.3
>>> > /home/hbase/hbase-0.94.2/lib/native/Linux-i386-32/libsnappy.so
>>> > /home/hbase/hbase-0.94.2/lib/native/Linux-i386-32/libsnappy.so.1
>>> > /home/hbase/hbase-0.94.2/lib/native/Linux-i386-32/libsnappy.so.1.1.3
>>> > /lib/x86_64-linux-gnu/libsnappy.so
>>> > /usr/lib/libsnappy.so
>>> > /usr/lib/libsnappy.so.1
>>> > /usr/lib/libsnappy.so.1.1.3
>>> > /usr/local/lib/libsnappy.so
>>> > /usr/local/lib/libsnappy.so.1
>>> > /usr/local/lib/libsnappy.so.1.1.3
>>> >
>>> > I tried to add this on my hbase-env.xml:
>>> > export
>>> > HBASE_LIBRARY_PATH=/home/hbase/hbase-0.94.2/lib/native/Linux-amd64-64
>>> >
>>> > Before I was trying with doing export on the command line directly
>>> > since it seems the hbase script is taking that into consideration too.
>>> >
>>> > I have not yet put the hbase.regionserver.codecs line since I still
>>> > need to use my cluster until I get snappy working. On the hbase/lib
>>> > directory I have snappy-java-1.0.3.2.jar.
>>> >
>>> >
>>> > Should snappy be installed within hbase? Or should it be in hadoop?
>>> > I'm not sure anymore.
>>> >
>>> > But it's still not working. So I tried the small code below:
>>> >
>>> > import java.util.StringTokenizer;
>>> >
>>> > public class Test
>>> > {
>>> >   static {
>>> >     try {
>>> >       System.loadLibrary("snappy");
>>> >       System.loadLibrary("hadoopsnappy");
>>> >       System.out.println ("Snappy native library is available");
>>> >     } catch (UnsatisfiedLinkError ex) {
>>> >         ex.printStackTrace();
>>> >     }
>>> >   }
>>> >
>>> >         public static void main (String [] args)
>>> >         {
>>> >                 System.out.println ("Coucou");
>>> > String property = System.getProperty("java.library.path");
>>> > StringTokenizer parser = new StringTokenizer(property, ";");
>>> > while (parser.hasMoreTokens()) {
>>> >     System.err.println(parser.nextToken());
>>> >     }
>>> >         }
>>> > }
>>> >
>>> >
>>> > This code is from org.apache.hadoop.io.compress.snappy.LoadSnappy.
>>> > The error I'm getting is java.lang.UnsatisfiedLinkError: no
>>> > hadoopsnappy in java.library.path.
>>> >
>>> > So the issue is not the snappy lib. It' there and working fine. The
>>> > issue is the hadoopsnappy lib which I don't have...
>>> >
>>> > I found it there: http://code.google.com/p/hadoop-snappy/
>>> >
>>> > So I have extracted it with svn checkout
>>> > http://hadoop-snappy.googlecode.com/svn/trunk/
>>> > hadoop-snappy-read-only, tried to built it with mvn package but it's
>>> > failing with something saying "cannot find -ljvm"
>>> >
>>> > So seems my challenge will be to build hadoop-snappy and not to
>>> > install snappy which is already there and working...
>>> >
>>> > JM
>>> >
>>> > 2012/12/3, surfer <surfer@crs4.it>:
>>> >> hope it helps. this is what I do on apache hadoop 1.0.x and hbase
>>> 0.92.y:
>>> >> in hbase-site.xml add:
>>> >>
>>> >> <property>
>>> >> <name>hbase.regionserver.codecs</name>
>>> >> <value>snappy</value>
>>> >> </property>
>>> >>
>>> >> copy that file into the hadoop conf directory.
>>> >>
>>> >> in hbase-env.sh:
>>> >> export HBASE_LIBRARY_PATH=/pathtoyourhadoop/lib/native/Linux-amd64-64
>>> >>
>>> >> ( In hbase-env.sh I set also HBASE_HOME, HBASE_CONF_DIR, HADOOP_HOME,
>>> >> HADOOP_CONF_DIR but I don't know if they contribute to make snappy
>>> >> working...)
>>> >>
>>> >> in /pathtoyourhadoop/lib/native/Linux-amd64-64 I have:
>>> >> libsnappy.a
>>> >> libsnappy.so
>>> >> libsnappy.so.1
>>> >> libsnappy.so.1.1.2
>>> >>
>>> >> good luck
>>> >> giovanni
>>> >>
>>> >>
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> On 12/02/2012 02:25 PM, Jean-Marc Spaggiari wrote:
>>> >>> So. I spent few hours on that yesterday with no luck.
>>> >>>
>>> >>> Here is what I did:
>>> >>> - Install the google tar, untared, configured, maked and installed
>>> >>> it.
>>> >>> - Copied the .so files all over my fs in the os lib dir,
>>> >>> HBase/lib/native and subdirs, Hadoop/lib/native and subdirs.
>>> >>> - Installed all debian packages with snappy in the name:
>>> >>> python-snappy, libsnappy-dev, libsnappy1, libsnappy-java
>>> >>>
>>> >>> But still exactly the same issue as above. And I don't have any
clue
>>> >>> where to dig. There is nothing on internet about that.
>>> >>>
>>> >>> Anyone faced that already while installing Snappy?
>>> >>>
>>> >>> JM
>>> >>>
>>> >>> 2012/12/1, Jean-Marc Spaggiari <jean-marc@spaggiari.org>:
>>> >>>> Sorry, I forgot to paste few maybe useful lines. I have the
lib in
>>> >>>> /usr/local/lib copied properly, and I have the HBASE_LIBRARY_PATH
>>> >>>> set
>>> >>>> correctly. Do I need to restart HBase to run this test?
>>> >>>>
>>> >>>> hbase@node3:~/hbase-0.94.2$ export
>>> HBASE_LIBRARY_PATH=/usr/local/lib/
>>> >>>> hbase@node3:~/hbase-0.94.2$ bin/hbase
>>> >>>> org.apache.hadoop.hbase.util.CompressionTest /tmp/test.txt snappy
>>> >>>> 12/12/01 18:55:29 INFO util.ChecksumType:
>>> >>>> org.apache.hadoop.util.PureJavaCrc32 not available.
>>> >>>> 12/12/01 18:55:29 INFO util.ChecksumType: Checksum can use
>>> >>>> java.util.zip.CRC32
>>> >>>> 12/12/01 18:55:29 INFO util.ChecksumType:
>>> >>>> org.apache.hadoop.util.PureJavaCrc32C not available.
>>> >>>> 12/12/01 18:55:29 DEBUG util.FSUtils: Creating
>>> >>>> file:/tmp/test.txtwith
>>> >>>> permission:rwxrwxrwx
>>> >>>> 12/12/01 18:55:29 WARN util.NativeCodeLoader: Unable to load
>>> >>>> native-hadoop library for your platform... using builtin-java
>>> >>>> classes
>>> >>>> where applicable
>>> >>>> 12/12/01 18:55:29 WARN metrics.SchemaConfigured: Could not
>>> >>>> determine
>>> >>>> table and column family of the HFile path /tmp/test.txt. Expecting
>>> >>>> at
>>> >>>> least 5 path components.
>>> >>>> 12/12/01 18:55:29 WARN snappy.LoadSnappy: Snappy native library
is
>>> >>>> available
>>> >>>> 12/12/01 18:55:29 WARN snappy.LoadSnappy: Snappy native library
not
>>> >>>> loaded
>>> >>>> Exception in thread "main" java.lang.RuntimeException: native
>>> >>>> snappy
>>> >>>> library not available
>>> >>>>    at
>>> >>>>
>>> org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:123)
>>> >>>>    at
>>> >>>>
>>> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:100)
>>> >>>>    at
>>> >>>>
>>> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:112)
>>> >>>>    at
>>> >>>>
>>> org.apache.hadoop.hbase.io.hfile.Compression$Algorithm.getCompressor(Compression.java:264)
>>> >>>>    at
>>> >>>>
>>> org.apache.hadoop.hbase.io.hfile.HFileBlock$Writer.<init>(HFileBlock.java:739)
>>> >>>>    at
>>> >>>>
>>> org.apache.hadoop.hbase.io.hfile.HFileWriterV2.finishInit(HFileWriterV2.java:127)
>>> >>>>    at
>>> >>>>
>>> org.apache.hadoop.hbase.io.hfile.HFileWriterV2.<init>(HFileWriterV2.java:118)
>>> >>>>    at
>>> >>>>
>>> org.apache.hadoop.hbase.io.hfile.HFileWriterV2$WriterFactoryV2.createWriter(HFileWriterV2.java:101)
>>> >>>>    at
>>> >>>>
>>> org.apache.hadoop.hbase.io.hfile.HFile$WriterFactory.create(HFile.java:394)
>>> >>>>    at
>>> >>>>
>>> org.apache.hadoop.hbase.util.CompressionTest.doSmokeTest(CompressionTest.java:108)
>>> >>>>    at
>>> >>>>
>>> org.apache.hadoop.hbase.util.CompressionTest.main(CompressionTest.java:138)
>>> >>>> hbase@node3:~/hbase-0.94.2$ ll /usr/local/lib/
>>> >>>> total 572
>>> >>>> -rw-r--r-- 1 root staff 391614 déc  1 18:33 libsnappy.a
>>> >>>> -rwxr-xr-x 1 root staff    957 déc  1 18:33 libsnappy.la
>>> >>>> lrwxrwxrwx 1 root staff     18 déc  1 18:33 libsnappy.so ->
>>> >>>> libsnappy.so.1.1.3
>>> >>>> lrwxrwxrwx 1 root staff     18 déc  1 18:33 libsnappy.so.1
->
>>> >>>> libsnappy.so.1.1.3
>>> >>>> -rwxr-xr-x 1 root staff 178210 déc  1 18:33 libsnappy.so.1.1.3
>>> >>>> drwxrwsr-x 4 root staff   4096 jui 13 10:06 python2.6
>>> >>>> drwxrwsr-x 4 root staff   4096 jui 13 10:06 python2.7
>>> >>>> hbase@node3:~/hbase-0.94.2$
>>> >>>>
>>> >>>>
>>> >>>> 2012/12/1, Jean-Marc Spaggiari <jean-marc@spaggiari.org>:
>>> >>>>> Hi,
>>> >>>>>
>>> >>>>> I'm currently using GZip and want to move to Snappy.
>>> >>>>>
>>> >>>>> I have downloaded the tar file, extracted, build, make install,
>>> >>>>> make
>>> >>>>> check, everything is working fine.
>>> >>>>>
>>> >>>>> However, I'm not able to get this working:
>>> >>>>> bin/hbase org.apache.hadoop.hbase.util.CompressionTest
>>> >>>>> /tmp/test.txt
>>> >>>>> snappy
>>> >>>>> 12/12/01 18:46:21 WARN snappy.LoadSnappy: Snappy native
library
>>> >>>>> not
>>> >>>>> loaded
>>> >>>>> Exception in thread "main" java.lang.RuntimeException: native
>>> >>>>> snappy
>>> >>>>> library not available
>>> >>>>>   at
>>> >>>>>
>>> org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:123)
>>> >>>>>
>>> >>>>> Sound like HBase is not able to find the native library.
How can I
>>> >>>>> tell HBase where the library is?
>>> >>>>>
>>> >>>>> Thanks,
>>> >>>>>
>>> >>>>> JM
>>> >>>>>
>>> >>
>>> >>
>>> >
>>>
>>
>>
>>
>> --
>> Kevin O'Dell
>> Customer Operations Engineer, Cloudera
>>
>
>
>
> --
> Kevin O'Dell
> Customer Operations Engineer, Cloudera
>

Mime
View raw message