hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From bharath vissapragada <bharathvissapragada1...@gmail.com>
Subject Re: Setting up Snappy compression in Hadoop
Date Thu, 02 Jan 2014 11:56:23 GMT
Your natives should be in LD_LIBRARY_PATH or java.library.path for hadoop
to pick them up. You can try adding export HADOOP_OPTS=$HADOOP_OPTS
-Djava.library.path=<Path to your natives lib> to hadoop-env.sh in TTs and
clients/gateways and restart TTs and give it another try. The reason its
working for Hbase is you are manually pointing HBASE_LIBRARY_PATH to the
natives. My guess is they are in a wrong location.


On Thu, Jan 2, 2014 at 5:07 PM, Amit Sela <amits@infolinks.com> wrote:

> I did everything mentioned in the link Ted mentioned, and the test
> actually works, but using Snappy for MapReduce map output compression still
> fails with "native snappy library not available".
>
>
> On Wed, Jan 1, 2014 at 6:37 PM, bharath vissapragada <
> bharathvissapragada1990@gmail.com> wrote:
>
>> Did you build it for your platform? You can do an "ldd" on the .so file
>> to check if the dependent libs are present. Also make sure you placed it in
>> the right directory for your platform (Linux-amd64-64 or Linux-i386-32)
>>
>>
>> On Wed, Jan 1, 2014 at 10:02 PM, Ted Yu <yuzhihong@gmail.com> wrote:
>>
>>> Please take a look at
>>> http://hbase.apache.org/book.html#snappy.compression
>>>
>>> Cheers
>>>
>>>
>>> On Wed, Jan 1, 2014 at 8:05 AM, Amit Sela <amits@infolinks.com> wrote:
>>>
>>>> Hi all,
>>>>
>>>> I'm running on Hadoop 1.0.4 and I'd like to use Snappy for map output
>>>> compression.
>>>> I'm adding the configurations:
>>>>
>>>> configuration.setBoolean("mapred.compress.map.output", true);
>>>> configuration.set("mapred.map.output.compression.codec",
>>>> "org.apache.hadoop.io.compress.SnappyCodec");
>>>>
>>>> And I've added libsnappy.so.1 to $HADOOP_HOME/lib/native/Linux-amd64-64/
>>>>
>>>> Still, all map tasks fail with "native snappy library not available".
>>>>
>>>> Could anyone elaborate on how to install Snappy for Hadoop ?
>>>>
>>>> Thanks,
>>>>
>>>> Amit.
>>>>
>>>
>>>
>>
>

Mime
View raw message