hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Marc Spaggiari <jean-m...@spaggiari.org>
Subject Re: How to install Snappy?
Date Mon, 03 Dec 2012 13:47:06 GMT
Ok....

This: http://code.google.com/p/hadoop-snappy/issues/detail?id=2 helped
me and my test program is now working. I'm able to load both
libraries. Fine.

But the CompressionTest is still not working.

What is very strange is that:
12/12/03 08:44:24 WARN snappy.LoadSnappy: Snappy native library is available
12/12/03 08:44:24 WARN snappy.LoadSnappy: Snappy native library not loaded

It's available, but not loaded.

But from the code:
  static {
    try {
      System.loadLibrary("snappy");
      System.loadLibrary("hadoopsnappy");
      LOG.warn("Snappy native library is available");
      AVAILABLE = true;
    } catch (UnsatisfiedLinkError ex) {
      //NOP
    }
    LOADED = AVAILABLE;
    if (LOADED) {
      LOG.info("Snappy native library loaded");
    } else {
      LOG.warn("Snappy native library not loaded");
    }
  }
If "Snappy native library is available" is displayed, that mean
AVAILABLE = true... And if AVAILABLE = true, then LOADED is set to
true and Snappy native library loaded must be displayed... But it's
not... How is this possible?

I have not expected Snappy installation to be such a challenge...

I will continue to dig and summarize the steps when I will be done (If
I'm able to finish...)

JM

2012/12/3, Jean-Marc Spaggiari <jean-marc@spaggiari.org>:
> Thanks all for your replies.
>
> So, to reply to all in one.
>
> I'm not using CD3. I'm using Hadoop  1.0.3 and HBase 0.94.2 directly
> from the JARs.
>
> Here are all the places where I have put the lib:
> /home/hadoop/hadoop-1.0.3/lib/native/Linux-amd64-64/libsnappy.so
> /home/hadoop/hadoop-1.0.3/lib/native/Linux-amd64-64/libsnappy.so.1
> /home/hadoop/hadoop-1.0.3/lib/native/Linux-amd64-64/libsnappy.so.1.1.3
> /home/hadoop/hadoop-1.0.3/lib/native/Linux-i386-32/libsnappy.so
> /home/hadoop/hadoop-1.0.3/lib/native/Linux-i386-32/libsnappy.so.1
> /home/hadoop/hadoop-1.0.3/lib/native/Linux-i386-32/libsnappy.so.1.1.3
> /home/hbase/hbase-0.94.2/lib/native/libsnappy.so
> /home/hbase/hbase-0.94.2/lib/native/libsnappy.so.1
> /home/hbase/hbase-0.94.2/lib/native/libsnappy.so.1.1.3
> /home/hbase/hbase-0.94.2/lib/native/Linux-amd64-64/libsnappy.so
> /home/hbase/hbase-0.94.2/lib/native/Linux-amd64-64/libsnappy.so.1
> /home/hbase/hbase-0.94.2/lib/native/Linux-amd64-64/libsnappy.so.1.1.3
> /home/hbase/hbase-0.94.2/lib/native/Linux-i386-32/libsnappy.so
> /home/hbase/hbase-0.94.2/lib/native/Linux-i386-32/libsnappy.so.1
> /home/hbase/hbase-0.94.2/lib/native/Linux-i386-32/libsnappy.so.1.1.3
> /lib/x86_64-linux-gnu/libsnappy.so
> /usr/lib/libsnappy.so
> /usr/lib/libsnappy.so.1
> /usr/lib/libsnappy.so.1.1.3
> /usr/local/lib/libsnappy.so
> /usr/local/lib/libsnappy.so.1
> /usr/local/lib/libsnappy.so.1.1.3
>
> I tried to add this on my hbase-env.xml:
> export
> HBASE_LIBRARY_PATH=/home/hbase/hbase-0.94.2/lib/native/Linux-amd64-64
>
> Before I was trying with doing export on the command line directly
> since it seems the hbase script is taking that into consideration too.
>
> I have not yet put the hbase.regionserver.codecs line since I still
> need to use my cluster until I get snappy working. On the hbase/lib
> directory I have snappy-java-1.0.3.2.jar.
>
>
> Should snappy be installed within hbase? Or should it be in hadoop?
> I'm not sure anymore.
>
> But it's still not working. So I tried the small code below:
>
> import java.util.StringTokenizer;
>
> public class Test
> {
>   static {
>     try {
>       System.loadLibrary("snappy");
>       System.loadLibrary("hadoopsnappy");
>       System.out.println ("Snappy native library is available");
>     } catch (UnsatisfiedLinkError ex) {
>         ex.printStackTrace();
>     }
>   }
>
>         public static void main (String [] args)
>         {
>                 System.out.println ("Coucou");
> String property = System.getProperty("java.library.path");
> StringTokenizer parser = new StringTokenizer(property, ";");
> while (parser.hasMoreTokens()) {
>     System.err.println(parser.nextToken());
>     }
>         }
> }
>
>
> This code is from org.apache.hadoop.io.compress.snappy.LoadSnappy.
> The error I'm getting is java.lang.UnsatisfiedLinkError: no
> hadoopsnappy in java.library.path.
>
> So the issue is not the snappy lib. It' there and working fine. The
> issue is the hadoopsnappy lib which I don't have...
>
> I found it there: http://code.google.com/p/hadoop-snappy/
>
> So I have extracted it with svn checkout
> http://hadoop-snappy.googlecode.com/svn/trunk/
> hadoop-snappy-read-only, tried to built it with mvn package but it's
> failing with something saying "cannot find -ljvm"
>
> So seems my challenge will be to build hadoop-snappy and not to
> install snappy which is already there and working...
>
> JM
>
> 2012/12/3, surfer <surfer@crs4.it>:
>> hope it helps. this is what I do on apache hadoop 1.0.x and hbase 0.92.y:
>> in hbase-site.xml add:
>>
>> <property>
>> <name>hbase.regionserver.codecs</name>
>> <value>snappy</value>
>> </property>
>>
>> copy that file into the hadoop conf directory.
>>
>> in hbase-env.sh:
>> export HBASE_LIBRARY_PATH=/pathtoyourhadoop/lib/native/Linux-amd64-64
>>
>> ( In hbase-env.sh I set also HBASE_HOME, HBASE_CONF_DIR, HADOOP_HOME,
>> HADOOP_CONF_DIR but I don't know if they contribute to make snappy
>> working...)
>>
>> in /pathtoyourhadoop/lib/native/Linux-amd64-64 I have:
>> libsnappy.a
>> libsnappy.so
>> libsnappy.so.1
>> libsnappy.so.1.1.2
>>
>> good luck
>> giovanni
>>
>>
>>
>>
>>
>>
>> On 12/02/2012 02:25 PM, Jean-Marc Spaggiari wrote:
>>> So. I spent few hours on that yesterday with no luck.
>>>
>>> Here is what I did:
>>> - Install the google tar, untared, configured, maked and installed it.
>>> - Copied the .so files all over my fs in the os lib dir,
>>> HBase/lib/native and subdirs, Hadoop/lib/native and subdirs.
>>> - Installed all debian packages with snappy in the name:
>>> python-snappy, libsnappy-dev, libsnappy1, libsnappy-java
>>>
>>> But still exactly the same issue as above. And I don't have any clue
>>> where to dig. There is nothing on internet about that.
>>>
>>> Anyone faced that already while installing Snappy?
>>>
>>> JM
>>>
>>> 2012/12/1, Jean-Marc Spaggiari <jean-marc@spaggiari.org>:
>>>> Sorry, I forgot to paste few maybe useful lines. I have the lib in
>>>> /usr/local/lib copied properly, and I have the HBASE_LIBRARY_PATH set
>>>> correctly. Do I need to restart HBase to run this test?
>>>>
>>>> hbase@node3:~/hbase-0.94.2$ export HBASE_LIBRARY_PATH=/usr/local/lib/
>>>> hbase@node3:~/hbase-0.94.2$ bin/hbase
>>>> org.apache.hadoop.hbase.util.CompressionTest /tmp/test.txt snappy
>>>> 12/12/01 18:55:29 INFO util.ChecksumType:
>>>> org.apache.hadoop.util.PureJavaCrc32 not available.
>>>> 12/12/01 18:55:29 INFO util.ChecksumType: Checksum can use
>>>> java.util.zip.CRC32
>>>> 12/12/01 18:55:29 INFO util.ChecksumType:
>>>> org.apache.hadoop.util.PureJavaCrc32C not available.
>>>> 12/12/01 18:55:29 DEBUG util.FSUtils: Creating file:/tmp/test.txtwith
>>>> permission:rwxrwxrwx
>>>> 12/12/01 18:55:29 WARN util.NativeCodeLoader: Unable to load
>>>> native-hadoop library for your platform... using builtin-java classes
>>>> where applicable
>>>> 12/12/01 18:55:29 WARN metrics.SchemaConfigured: Could not determine
>>>> table and column family of the HFile path /tmp/test.txt. Expecting at
>>>> least 5 path components.
>>>> 12/12/01 18:55:29 WARN snappy.LoadSnappy: Snappy native library is
>>>> available
>>>> 12/12/01 18:55:29 WARN snappy.LoadSnappy: Snappy native library not
>>>> loaded
>>>> Exception in thread "main" java.lang.RuntimeException: native snappy
>>>> library not available
>>>> 	at
>>>> org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:123)
>>>> 	at
>>>> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:100)
>>>> 	at
>>>> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:112)
>>>> 	at
>>>> org.apache.hadoop.hbase.io.hfile.Compression$Algorithm.getCompressor(Compression.java:264)
>>>> 	at
>>>> org.apache.hadoop.hbase.io.hfile.HFileBlock$Writer.<init>(HFileBlock.java:739)
>>>> 	at
>>>> org.apache.hadoop.hbase.io.hfile.HFileWriterV2.finishInit(HFileWriterV2.java:127)
>>>> 	at
>>>> org.apache.hadoop.hbase.io.hfile.HFileWriterV2.<init>(HFileWriterV2.java:118)
>>>> 	at
>>>> org.apache.hadoop.hbase.io.hfile.HFileWriterV2$WriterFactoryV2.createWriter(HFileWriterV2.java:101)
>>>> 	at
>>>> org.apache.hadoop.hbase.io.hfile.HFile$WriterFactory.create(HFile.java:394)
>>>> 	at
>>>> org.apache.hadoop.hbase.util.CompressionTest.doSmokeTest(CompressionTest.java:108)
>>>> 	at
>>>> org.apache.hadoop.hbase.util.CompressionTest.main(CompressionTest.java:138)
>>>> hbase@node3:~/hbase-0.94.2$ ll /usr/local/lib/
>>>> total 572
>>>> -rw-r--r-- 1 root staff 391614 déc  1 18:33 libsnappy.a
>>>> -rwxr-xr-x 1 root staff    957 déc  1 18:33 libsnappy.la
>>>> lrwxrwxrwx 1 root staff     18 déc  1 18:33 libsnappy.so ->
>>>> libsnappy.so.1.1.3
>>>> lrwxrwxrwx 1 root staff     18 déc  1 18:33 libsnappy.so.1 ->
>>>> libsnappy.so.1.1.3
>>>> -rwxr-xr-x 1 root staff 178210 déc  1 18:33 libsnappy.so.1.1.3
>>>> drwxrwsr-x 4 root staff   4096 jui 13 10:06 python2.6
>>>> drwxrwsr-x 4 root staff   4096 jui 13 10:06 python2.7
>>>> hbase@node3:~/hbase-0.94.2$
>>>>
>>>>
>>>> 2012/12/1, Jean-Marc Spaggiari <jean-marc@spaggiari.org>:
>>>>> Hi,
>>>>>
>>>>> I'm currently using GZip and want to move to Snappy.
>>>>>
>>>>> I have downloaded the tar file, extracted, build, make install, make
>>>>> check, everything is working fine.
>>>>>
>>>>> However, I'm not able to get this working:
>>>>> bin/hbase org.apache.hadoop.hbase.util.CompressionTest /tmp/test.txt
>>>>> snappy
>>>>> 12/12/01 18:46:21 WARN snappy.LoadSnappy: Snappy native library not
>>>>> loaded
>>>>> Exception in thread "main" java.lang.RuntimeException: native snappy
>>>>> library not available
>>>>> 	at
>>>>> org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:123)
>>>>>
>>>>> Sound like HBase is not able to find the native library. How can I
>>>>> tell HBase where the library is?
>>>>>
>>>>> Thanks,
>>>>>
>>>>> JM
>>>>>
>>
>>
>

Mime
View raw message