hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From java8964 <java8...@hotmail.com>
Subject RE: Setting up Snappy compression in Hadoop
Date Thu, 02 Jan 2014 16:02:29 GMT
If you really confirmed that libsnappy.so.1 is in the correct location, and being loaded into
java library path, working in your test program, but still didn't work in MR, there is one
another possibility which was puzzling me before.
How do you get the libhadoop.so in your hadoop environment? Did you compile it by yourself,
or is it  from some vendors?
You want to make sure the Java native methods of invoking snappy is also being compiled and
available in libhadoop.so.
For example, the following command will prove it:
$ nm ./libhadoop.so | grep snappy00000000000035c0 T Java_org_apache_hadoop_io_compress_snappy_SnappyCompressor_compressBytesDirect0000000000003960
T Java_org_apache_hadoop_io_compress_snappy_SnappyCompressor_initIDs0000000000003bb0 T Java_org_apache_hadoop_io_compress_snappy_SnappyDecompressor_decompressBytesDirect0000000000003f60
T Java_org_apache_hadoop_io_compress_snappy_SnappyDecompressor_initIDs0000000000206cf0 b dlsym_snappy_compress0000000000206d20
b dlsym_snappy_uncompress
Without these Java native methods being compiled and available in the libhadoop.so, MR runtime
will also complain that "native snappy library not available".
Date: Thu, 2 Jan 2014 13:37:46 +0200
Subject: Re: Setting up Snappy compression in Hadoop
From: amits@infolinks.com
To: user@hadoop.apache.org

I did everything mentioned in the link Ted mentioned, and the test actually works, but using
Snappy for MapReduce map output compression still fails with "native snappy library not available".

On Wed, Jan 1, 2014 at 6:37 PM, bharath vissapragada <bharathvissapragada1990@gmail.com>

Did you build it for your platform? You can do an "ldd" on the .so file to check if the dependent
libs are present. Also make sure you placed it in the right directory for your platform (Linux-amd64-64
or Linux-i386-32)

On Wed, Jan 1, 2014 at 10:02 PM, Ted Yu <yuzhihong@gmail.com> wrote:

Please take a look at http://hbase.apache.org/book.html#snappy.compression

On Wed, Jan 1, 2014 at 8:05 AM, Amit Sela <amits@infolinks.com> wrote:

Hi all, 
I'm running on Hadoop 1.0.4 and I'd like to use Snappy for map output compression.I'm adding
the configurations:
configuration.setBoolean("mapred.compress.map.output", true);

configuration.set("mapred.map.output.compression.codec", "org.apache.hadoop.io.compress.SnappyCodec");

And I've added libsnappy.so.1 to $HADOOP_HOME/lib/native/Linux-amd64-64/

Still, all map tasks fail with "native snappy library not available".
Could anyone elaborate on how to install Snappy for Hadoop ?


View raw message