hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Marc Spaggiari <jean-m...@spaggiari.org>
Subject Re: How to install Snappy?
Date Mon, 03 Dec 2012 15:15:09 GMT
Ok, I got it!!!!

I had to copy the hadoop native libs into the hbase native libs directory!

Now I get a SUCCESS when I'm doint the CompressionTest...

I'm not 100% sure that it's the only think which was missing because I
have done so many modifications in the last 3 days...

So I will start from a blank 0.94.3 jar and re-do all the steps to
make sure it's just the native libs which need to be copied.

I was close to surrender ;)

JM

2012/12/3, Jean-Marc Spaggiari <jean-marc@spaggiari.org>:
> Hi Kevin,
>
> Thanks for the clarification.
>
> No, it's not what I'm seeing.
>
> Here is what I'm getting:
>
> 12/12/03 09:40:42 WARN snappy.LoadSnappy: Snappy native library is
> available
> 12/12/03 09:40:42 WARN snappy.LoadSnappy: Snappy native library not loaded
> Exception in thread "main" java.lang.RuntimeException: native snappy
> library not available
> 	at
> org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:123)
> 	at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:100)
> 	at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:112)
> 	at
> org.apache.hadoop.hbase.io.hfile.Compression$Algorithm.getCompressor(Compression.java:264)
> 	at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$Writer.<init>(HFileBlock.java:739)
> 	at
> org.apache.hadoop.hbase.io.hfile.HFileWriterV2.finishInit(HFileWriterV2.java:127)
> 	at
> org.apache.hadoop.hbase.io.hfile.HFileWriterV2.<init>(HFileWriterV2.java:118)
> 	at
> org.apache.hadoop.hbase.io.hfile.HFileWriterV2$WriterFactoryV2.createWriter(HFileWriterV2.java:101)
> 	at
> org.apache.hadoop.hbase.io.hfile.HFile$WriterFactory.create(HFile.java:394)
> 	at
> org.apache.hadoop.hbase.util.CompressionTest.doSmokeTest(CompressionTest.java:108)
> 	at
> org.apache.hadoop.hbase.util.CompressionTest.main(CompressionTest.java:138)
>
> The most disturbing part is this line:
> 12/12/03 09:40:42 WARN snappy.LoadSnappy: Snappy native library is
> available
>
> Followed by this one:
> Exception in thread "main" java.lang.RuntimeException: native snappy
> library not available
>
> Is it available? Or is it not available?
>
> I looked in the code and I have no idea why the 2nd one is raised.
>
> The code I'm looking at is on the hadoop-snappy site, but the one I
> have on my server is in the hadoop the hadoop-core-1.0.3.jar file. So
> maybe that the issue and they are different?
>
> I built the hadoop-snappy-0.0.1-SNAPSHOT.jar file too. I placed it on
> the lib folder and made sure it was taken first, but still not
> working.
>
> So far I think I will stay with GZip until Snappy is integrated on the
> HBase files...
>
> JM
>
> 2012/12/3, Kevin O'dell <kevin.odell@cloudera.com>:
>> Hey JM,
>>
>>   Sorry for the quick message earlier.  I tracked down the JIRA I was
>> referring to: https://issues.apache.org/jira/browse/HBASE-7080
>>
>> Does this look like what you are seeing in Compression test?
>>
>> On Mon, Dec 3, 2012 at 9:09 AM, Kevin O'dell
>> <kevin.odell@cloudera.com>wrote:
>>
>>> There is a compression test JIRA right now.  What are you seeing?
>>>
>>>
>>> On Mon, Dec 3, 2012 at 8:47 AM, Jean-Marc Spaggiari <
>>> jean-marc@spaggiari.org> wrote:
>>>
>>>> Ok....
>>>>
>>>> This: http://code.google.com/p/hadoop-snappy/issues/detail?id=2 helped
>>>> me and my test program is now working. I'm able to load both
>>>> libraries. Fine.
>>>>
>>>> But the CompressionTest is still not working.
>>>>
>>>> What is very strange is that:
>>>> 12/12/03 08:44:24 WARN snappy.LoadSnappy: Snappy native library is
>>>> available
>>>> 12/12/03 08:44:24 WARN snappy.LoadSnappy: Snappy native library not
>>>> loaded
>>>>
>>>> It's available, but not loaded.
>>>>
>>>> But from the code:
>>>>   static {
>>>>     try {
>>>>       System.loadLibrary("snappy");
>>>>       System.loadLibrary("hadoopsnappy");
>>>>       LOG.warn("Snappy native library is available");
>>>>       AVAILABLE = true;
>>>>     } catch (UnsatisfiedLinkError ex) {
>>>>       //NOP
>>>>     }
>>>>     LOADED = AVAILABLE;
>>>>     if (LOADED) {
>>>>       LOG.info("Snappy native library loaded");
>>>>     } else {
>>>>       LOG.warn("Snappy native library not loaded");
>>>>     }
>>>>   }
>>>> If "Snappy native library is available" is displayed, that mean
>>>> AVAILABLE = true... And if AVAILABLE = true, then LOADED is set to
>>>> true and Snappy native library loaded must be displayed... But it's
>>>> not... How is this possible?
>>>>
>>>> I have not expected Snappy installation to be such a challenge...
>>>>
>>>> I will continue to dig and summarize the steps when I will be done (If
>>>> I'm able to finish...)
>>>>
>>>> JM
>>>>
>>>> 2012/12/3, Jean-Marc Spaggiari <jean-marc@spaggiari.org>:
>>>> > Thanks all for your replies.
>>>> >
>>>> > So, to reply to all in one.
>>>> >
>>>> > I'm not using CD3. I'm using Hadoop  1.0.3 and HBase 0.94.2 directly
>>>> > from the JARs.
>>>> >
>>>> > Here are all the places where I have put the lib:
>>>> > /home/hadoop/hadoop-1.0.3/lib/native/Linux-amd64-64/libsnappy.so
>>>> > /home/hadoop/hadoop-1.0.3/lib/native/Linux-amd64-64/libsnappy.so.1
>>>> > /home/hadoop/hadoop-1.0.3/lib/native/Linux-amd64-64/libsnappy.so.1.1.3
>>>> > /home/hadoop/hadoop-1.0.3/lib/native/Linux-i386-32/libsnappy.so
>>>> > /home/hadoop/hadoop-1.0.3/lib/native/Linux-i386-32/libsnappy.so.1
>>>> > /home/hadoop/hadoop-1.0.3/lib/native/Linux-i386-32/libsnappy.so.1.1.3
>>>> > /home/hbase/hbase-0.94.2/lib/native/libsnappy.so
>>>> > /home/hbase/hbase-0.94.2/lib/native/libsnappy.so.1
>>>> > /home/hbase/hbase-0.94.2/lib/native/libsnappy.so.1.1.3
>>>> > /home/hbase/hbase-0.94.2/lib/native/Linux-amd64-64/libsnappy.so
>>>> > /home/hbase/hbase-0.94.2/lib/native/Linux-amd64-64/libsnappy.so.1
>>>> > /home/hbase/hbase-0.94.2/lib/native/Linux-amd64-64/libsnappy.so.1.1.3
>>>> > /home/hbase/hbase-0.94.2/lib/native/Linux-i386-32/libsnappy.so
>>>> > /home/hbase/hbase-0.94.2/lib/native/Linux-i386-32/libsnappy.so.1
>>>> > /home/hbase/hbase-0.94.2/lib/native/Linux-i386-32/libsnappy.so.1.1.3
>>>> > /lib/x86_64-linux-gnu/libsnappy.so
>>>> > /usr/lib/libsnappy.so
>>>> > /usr/lib/libsnappy.so.1
>>>> > /usr/lib/libsnappy.so.1.1.3
>>>> > /usr/local/lib/libsnappy.so
>>>> > /usr/local/lib/libsnappy.so.1
>>>> > /usr/local/lib/libsnappy.so.1.1.3
>>>> >
>>>> > I tried to add this on my hbase-env.xml:
>>>> > export
>>>> > HBASE_LIBRARY_PATH=/home/hbase/hbase-0.94.2/lib/native/Linux-amd64-64
>>>> >
>>>> > Before I was trying with doing export on the command line directly
>>>> > since it seems the hbase script is taking that into consideration
>>>> > too.
>>>> >
>>>> > I have not yet put the hbase.regionserver.codecs line since I still
>>>> > need to use my cluster until I get snappy working. On the hbase/lib
>>>> > directory I have snappy-java-1.0.3.2.jar.
>>>> >
>>>> >
>>>> > Should snappy be installed within hbase? Or should it be in hadoop?
>>>> > I'm not sure anymore.
>>>> >
>>>> > But it's still not working. So I tried the small code below:
>>>> >
>>>> > import java.util.StringTokenizer;
>>>> >
>>>> > public class Test
>>>> > {
>>>> >   static {
>>>> >     try {
>>>> >       System.loadLibrary("snappy");
>>>> >       System.loadLibrary("hadoopsnappy");
>>>> >       System.out.println ("Snappy native library is available");
>>>> >     } catch (UnsatisfiedLinkError ex) {
>>>> >         ex.printStackTrace();
>>>> >     }
>>>> >   }
>>>> >
>>>> >         public static void main (String [] args)
>>>> >         {
>>>> >                 System.out.println ("Coucou");
>>>> > String property = System.getProperty("java.library.path");
>>>> > StringTokenizer parser = new StringTokenizer(property, ";");
>>>> > while (parser.hasMoreTokens()) {
>>>> >     System.err.println(parser.nextToken());
>>>> >     }
>>>> >         }
>>>> > }
>>>> >
>>>> >
>>>> > This code is from org.apache.hadoop.io.compress.snappy.LoadSnappy.
>>>> > The error I'm getting is java.lang.UnsatisfiedLinkError: no
>>>> > hadoopsnappy in java.library.path.
>>>> >
>>>> > So the issue is not the snappy lib. It' there and working fine. The
>>>> > issue is the hadoopsnappy lib which I don't have...
>>>> >
>>>> > I found it there: http://code.google.com/p/hadoop-snappy/
>>>> >
>>>> > So I have extracted it with svn checkout
>>>> > http://hadoop-snappy.googlecode.com/svn/trunk/
>>>> > hadoop-snappy-read-only, tried to built it with mvn package but it's
>>>> > failing with something saying "cannot find -ljvm"
>>>> >
>>>> > So seems my challenge will be to build hadoop-snappy and not to
>>>> > install snappy which is already there and working...
>>>> >
>>>> > JM
>>>> >
>>>> > 2012/12/3, surfer <surfer@crs4.it>:
>>>> >> hope it helps. this is what I do on apache hadoop 1.0.x and hbase
>>>> 0.92.y:
>>>> >> in hbase-site.xml add:
>>>> >>
>>>> >> <property>
>>>> >> <name>hbase.regionserver.codecs</name>
>>>> >> <value>snappy</value>
>>>> >> </property>
>>>> >>
>>>> >> copy that file into the hadoop conf directory.
>>>> >>
>>>> >> in hbase-env.sh:
>>>> >> export
>>>> >> HBASE_LIBRARY_PATH=/pathtoyourhadoop/lib/native/Linux-amd64-64
>>>> >>
>>>> >> ( In hbase-env.sh I set also HBASE_HOME, HBASE_CONF_DIR,
>>>> >> HADOOP_HOME,
>>>> >> HADOOP_CONF_DIR but I don't know if they contribute to make snappy
>>>> >> working...)
>>>> >>
>>>> >> in /pathtoyourhadoop/lib/native/Linux-amd64-64 I have:
>>>> >> libsnappy.a
>>>> >> libsnappy.so
>>>> >> libsnappy.so.1
>>>> >> libsnappy.so.1.1.2
>>>> >>
>>>> >> good luck
>>>> >> giovanni
>>>> >>
>>>> >>
>>>> >>
>>>> >>
>>>> >>
>>>> >>
>>>> >> On 12/02/2012 02:25 PM, Jean-Marc Spaggiari wrote:
>>>> >>> So. I spent few hours on that yesterday with no luck.
>>>> >>>
>>>> >>> Here is what I did:
>>>> >>> - Install the google tar, untared, configured, maked and installed
>>>> >>> it.
>>>> >>> - Copied the .so files all over my fs in the os lib dir,
>>>> >>> HBase/lib/native and subdirs, Hadoop/lib/native and subdirs.
>>>> >>> - Installed all debian packages with snappy in the name:
>>>> >>> python-snappy, libsnappy-dev, libsnappy1, libsnappy-java
>>>> >>>
>>>> >>> But still exactly the same issue as above. And I don't have
any
>>>> >>> clue
>>>> >>> where to dig. There is nothing on internet about that.
>>>> >>>
>>>> >>> Anyone faced that already while installing Snappy?
>>>> >>>
>>>> >>> JM
>>>> >>>
>>>> >>> 2012/12/1, Jean-Marc Spaggiari <jean-marc@spaggiari.org>:
>>>> >>>> Sorry, I forgot to paste few maybe useful lines. I have
the lib in
>>>> >>>> /usr/local/lib copied properly, and I have the HBASE_LIBRARY_PATH
>>>> >>>> set
>>>> >>>> correctly. Do I need to restart HBase to run this test?
>>>> >>>>
>>>> >>>> hbase@node3:~/hbase-0.94.2$ export
>>>> HBASE_LIBRARY_PATH=/usr/local/lib/
>>>> >>>> hbase@node3:~/hbase-0.94.2$ bin/hbase
>>>> >>>> org.apache.hadoop.hbase.util.CompressionTest /tmp/test.txt
snappy
>>>> >>>> 12/12/01 18:55:29 INFO util.ChecksumType:
>>>> >>>> org.apache.hadoop.util.PureJavaCrc32 not available.
>>>> >>>> 12/12/01 18:55:29 INFO util.ChecksumType: Checksum can use
>>>> >>>> java.util.zip.CRC32
>>>> >>>> 12/12/01 18:55:29 INFO util.ChecksumType:
>>>> >>>> org.apache.hadoop.util.PureJavaCrc32C not available.
>>>> >>>> 12/12/01 18:55:29 DEBUG util.FSUtils: Creating
>>>> >>>> file:/tmp/test.txtwith
>>>> >>>> permission:rwxrwxrwx
>>>> >>>> 12/12/01 18:55:29 WARN util.NativeCodeLoader: Unable to
load
>>>> >>>> native-hadoop library for your platform... using builtin-java
>>>> >>>> classes
>>>> >>>> where applicable
>>>> >>>> 12/12/01 18:55:29 WARN metrics.SchemaConfigured: Could not
>>>> >>>> determine
>>>> >>>> table and column family of the HFile path /tmp/test.txt.
Expecting
>>>> >>>> at
>>>> >>>> least 5 path components.
>>>> >>>> 12/12/01 18:55:29 WARN snappy.LoadSnappy: Snappy native
library is
>>>> >>>> available
>>>> >>>> 12/12/01 18:55:29 WARN snappy.LoadSnappy: Snappy native
library
>>>> >>>> not
>>>> >>>> loaded
>>>> >>>> Exception in thread "main" java.lang.RuntimeException: native
>>>> >>>> snappy
>>>> >>>> library not available
>>>> >>>>    at
>>>> >>>>
>>>> org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:123)
>>>> >>>>    at
>>>> >>>>
>>>> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:100)
>>>> >>>>    at
>>>> >>>>
>>>> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:112)
>>>> >>>>    at
>>>> >>>>
>>>> org.apache.hadoop.hbase.io.hfile.Compression$Algorithm.getCompressor(Compression.java:264)
>>>> >>>>    at
>>>> >>>>
>>>> org.apache.hadoop.hbase.io.hfile.HFileBlock$Writer.<init>(HFileBlock.java:739)
>>>> >>>>    at
>>>> >>>>
>>>> org.apache.hadoop.hbase.io.hfile.HFileWriterV2.finishInit(HFileWriterV2.java:127)
>>>> >>>>    at
>>>> >>>>
>>>> org.apache.hadoop.hbase.io.hfile.HFileWriterV2.<init>(HFileWriterV2.java:118)
>>>> >>>>    at
>>>> >>>>
>>>> org.apache.hadoop.hbase.io.hfile.HFileWriterV2$WriterFactoryV2.createWriter(HFileWriterV2.java:101)
>>>> >>>>    at
>>>> >>>>
>>>> org.apache.hadoop.hbase.io.hfile.HFile$WriterFactory.create(HFile.java:394)
>>>> >>>>    at
>>>> >>>>
>>>> org.apache.hadoop.hbase.util.CompressionTest.doSmokeTest(CompressionTest.java:108)
>>>> >>>>    at
>>>> >>>>
>>>> org.apache.hadoop.hbase.util.CompressionTest.main(CompressionTest.java:138)
>>>> >>>> hbase@node3:~/hbase-0.94.2$ ll /usr/local/lib/
>>>> >>>> total 572
>>>> >>>> -rw-r--r-- 1 root staff 391614 déc  1 18:33 libsnappy.a
>>>> >>>> -rwxr-xr-x 1 root staff    957 déc  1 18:33 libsnappy.la
>>>> >>>> lrwxrwxrwx 1 root staff     18 déc  1 18:33 libsnappy.so
->
>>>> >>>> libsnappy.so.1.1.3
>>>> >>>> lrwxrwxrwx 1 root staff     18 déc  1 18:33 libsnappy.so.1
->
>>>> >>>> libsnappy.so.1.1.3
>>>> >>>> -rwxr-xr-x 1 root staff 178210 déc  1 18:33 libsnappy.so.1.1.3
>>>> >>>> drwxrwsr-x 4 root staff   4096 jui 13 10:06 python2.6
>>>> >>>> drwxrwsr-x 4 root staff   4096 jui 13 10:06 python2.7
>>>> >>>> hbase@node3:~/hbase-0.94.2$
>>>> >>>>
>>>> >>>>
>>>> >>>> 2012/12/1, Jean-Marc Spaggiari <jean-marc@spaggiari.org>:
>>>> >>>>> Hi,
>>>> >>>>>
>>>> >>>>> I'm currently using GZip and want to move to Snappy.
>>>> >>>>>
>>>> >>>>> I have downloaded the tar file, extracted, build, make
install,
>>>> >>>>> make
>>>> >>>>> check, everything is working fine.
>>>> >>>>>
>>>> >>>>> However, I'm not able to get this working:
>>>> >>>>> bin/hbase org.apache.hadoop.hbase.util.CompressionTest
>>>> >>>>> /tmp/test.txt
>>>> >>>>> snappy
>>>> >>>>> 12/12/01 18:46:21 WARN snappy.LoadSnappy: Snappy native
library
>>>> >>>>> not
>>>> >>>>> loaded
>>>> >>>>> Exception in thread "main" java.lang.RuntimeException:
native
>>>> >>>>> snappy
>>>> >>>>> library not available
>>>> >>>>>   at
>>>> >>>>>
>>>> org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:123)
>>>> >>>>>
>>>> >>>>> Sound like HBase is not able to find the native library.
How can
>>>> >>>>> I
>>>> >>>>> tell HBase where the library is?
>>>> >>>>>
>>>> >>>>> Thanks,
>>>> >>>>>
>>>> >>>>> JM
>>>> >>>>>
>>>> >>
>>>> >>
>>>> >
>>>>
>>>
>>>
>>>
>>> --
>>> Kevin O'Dell
>>> Customer Operations Engineer, Cloudera
>>>
>>
>>
>>
>> --
>> Kevin O'Dell
>> Customer Operations Engineer, Cloudera
>>
>

Mime
View raw message