Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (nike.apache.org: domain of peter.s.naudus@gmail.com
 designates 209.85.212.41 as permitted sender)
Content-Type: text/plain; charset=utf-8; format=flowed; delsp=yes
To: user@hbase.apache.org, "Nathaniel Cook" <nvcook42@gmail.com>
Subject: Re: java.io.IOException: Compression algorithm 'snappy' previously
 failed test
Reply-To: pnaudus@dataraker.com
References: <op.waescyeemrx7t6@localhost.localdomain>
 <CADcMMgENW4CceT-W3E4F3GnGsFhSJZ6_e95Ldkh+eXfpWTzpRA@mail.gmail.com>
 <op.wagbxvrpmrx7t6@localhost.localdomain>
 <CAJLiJmO_8LLTJ23h6+Us0cbfR2ZJH6FvT60_LSkn=eTh2CiGoQ@mail.gmail.com>
Date: Mon, 23 Apr 2012 11:23:50 -0400
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
From: "Peter Naudus" <pnaudus@dataraker.com>
Organization: DataRaker
Message-ID: <op.wc7410xjmrx7t6@localhost.localdomain>
In-Reply-To: 
 <CAJLiJmO_8LLTJ23h6+Us0cbfR2ZJH6FvT60_LSkn=eTh2CiGoQ@mail.gmail.com>
User-Agent: Opera Mail/11.62 (Linux)

Hi Nathaniel,

My memory's a little hazy (we ended going back to CDH3), but I believe the  
key to fixing this problem for me was the following log message:

> WARN util.NativeCodeLoader: Unable to load native-hadoop library for
> your platform... using builtin-java classes where applicable

I had set HBASE_LIBRARY_PATH to hadoop's library path since that was where  
snappy was. Once I set the path ( or linked created symbolic links ) so  
that HBASE_LIBRARY_PATH included both HBase's and Hadoop's the snappy  
issue was resolved.

Also, in general, I ended up having a lot better luck with the RPMs  
instead of the tarballs.

~ Peter

On Mon, 23 Apr 2012 11:01:19 -0400, Nathaniel Cook <nvcook42@gmail.com>  
wrote:

> Was there any resolution to this? I am experiencing the same issue.
>
> Nathaniel
>
> On Wed, Feb 29, 2012 at 10:52 AM, Peter Naudus <pnaudus@dataraker.com>  
> wrote:
>> Thanks for your help :)
>>
>> To make sure I manually set LD_LIBRARY_PATH, LIBRARY_PATH, and
>> HBASE_LIBRARY_PATH
>>
>> bash-3.2$ export
>> LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/dataraker/software/cdh4/hadoop-0.23.0-cdh4b1/lib/native
>> bash-3.2$ export
>> LIBRARY_PATH=$LIBRARY_PATH:/opt/dataraker/software/cdh4/hadoop-0.23.0-cdh4b1/lib/native
>> bash-3.2$ export
>> HBASE_LIBRARY_PATH=/opt/dataraker/software/cdh4/hadoop-0.23.0-cdh4b1/lib/native
>>
>> But running the compression test failed with "native snappy library not
>> available"
>>
>> bash-3.2$ ./hbase org.apache.hadoop.hbase.util.CompressionTest
>> file:///tmp/test.txt snappy
>> log4j:WARN No appenders could be found for logger
>> (org.apache.hadoop.conf.Configuration).
>> log4j:WARN Please initialize the log4j system properly.
>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for
>> more info.
>> Exception in thread "main" java.lang.RuntimeException: native snappy  
>> library
>> not available
>>        at
>> org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:121)
>>        at
>> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:104)
>>        at
>> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:118)
>>        at
>> org.apache.hadoop.hbase.io.hfile.Compression$Algorithm.getCompressor(Compression.java:236)
>>        at
>> org.apache.hadoop.hbase.io.hfile.HFileBlock$Writer.<init>(HFileBlock.java:588)
>>        at
>> org.apache.hadoop.hbase.io.hfile.HFileWriterV2.finishInit(HFileWriterV2.java:178)
>>        at
>> org.apache.hadoop.hbase.io.hfile.HFileWriterV2.<init>(HFileWriterV2.java:150)
>>        at
>> org.apache.hadoop.hbase.io.hfile.HFileWriterV2.<init>(HFileWriterV2.java:140)
>>        at
>> org.apache.hadoop.hbase.io.hfile.HFileWriterV2$WriterFactoryV2.createWriter(HFileWriterV2.java:104)
>>        at
>> org.apache.hadoop.hbase.util.CompressionTest.doSmokeTest(CompressionTest.java:108)
>>        at
>> org.apache.hadoop.hbase.util.CompressionTest.main(CompressionTest.java:137)
>>
>> I verified that libsnappy is indeed, installed
>>
>> bash-3.2$ ls -al $HBASE_LIBRARY_PATH
>> total 1412
>> drwxr-xr-x 2 1106 592   4096 Feb 11 01:06 .
>> drwxr-xr-x 3 1106 592   4096 Feb 11 01:06 ..
>> -rw-r--r-- 1 1106 592 616862 Feb 11 01:06 libhadoop.a
>> -rwxr-xr-x 1 1106 592   1051 Feb 11 01:06 libhadoop.la
>> lrwxrwxrwx 1 1106 592     18 Feb 27 18:12 libhadoop.so ->  
>> libhadoop.so.1.0.0
>> lrwxrwxrwx 1 1106 592     18 Feb 27 18:12 libhadoop.so.1 ->
>> libhadoop.so.1.0.0
>> -rwxr-xr-x 1 1106 592 340361 Feb 11 01:06 libhadoop.so.1.0.0
>> -rw-r--r-- 1 1106 592 184418 Feb 11 01:06 libhdfs.a
>> -rwxr-xr-x 1 1106 592   1034 Feb 11 01:06 libhdfs.la
>> lrwxrwxrwx 1 1106 592     16 Feb 27 18:12 libhdfs.so -> libhdfs.so.0.0.0
>> lrwxrwxrwx 1 1106 592     16 Feb 27 18:12 libhdfs.so.0 ->  
>> libhdfs.so.0.0.0
>> -rwxr-xr-x 1 1106 592 125455 Feb 11 01:06 libhdfs.so.0.0.0
>> -rw-r--r-- 1 1106 592  37392 Feb 11 01:06 libsnappy.a
>> lrwxrwxrwx 1 1106 592     18 Feb 27 18:12 libsnappy.so ->  
>> libsnappy.so.1.1.1
>> lrwxrwxrwx 1 1106 592     18 Feb 27 18:12 libsnappy.so.1 ->
>> libsnappy.so.1.1.1
>> -rw-r--r-- 1 1106 592  26824 Feb 11 01:06 libsnappy.so.1.1.1
>>
>> Just for grins and giggles I re-ran this as root
>>
>> In addition to the Exception mentioned above, I also got following  
>> Warning:
>>        WARN util.NativeCodeLoader: Unable to load native-hadoop library  
>> for
>> your platform... using builtin-java classes where applicable
>>
>> Any ideas?
>>
>>
>> On Tue, 28 Feb 2012 20:02:38 -0500, Stack <stack@duboce.net> wrote:
>>
>>> On Tue, Feb 28, 2012 at 1:52 PM, Peter Naudus <pnaudus@dataraker.com>
>>> wrote:
>>>>
>>>> What else can I do to fix / diagnose this problem?
>>>>
>>>
>>> Does our little compression tool help?
>>> http://hbase.apache.org/book.html#compression.test
>>>
>>> St.Ack
>>
>>
>>
>> --