hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From java8964 java8964 <java8...@hotmail.com>
Subject enable snappy on hadoop 1.1.1
Date Fri, 04 Oct 2013 19:44:34 GMT
I am using hadoop 1.1.1. I want to test to see the snappy compression with hadoop, but I have
some problems to make it work on my Linux environment.
I am using opensuse 12.3 x86_64. 
First, when I tried to enable snappy in hadoop 1.1.1 by:
            conf.setBoolean("mapred.compress.map.output", true);            conf.set("mapred.output.compression.type",
"RECORD");            conf.set("mapred.map.output.compression.codec", "org.apache.hadoop.io.compress.SnappyCodec");
I got the following error in my test MR job:
Exception in thread "main" java.lang.RuntimeException: native snappy library not available
So I download the snappy 1.1.0 from https://code.google.com/p/snappy/, compile it and install
it successfully under /opt/snappy-1.1.0, and then I link the /opt/snappy-1.1.0/lib64/libsnappy.so
to /user/lib64/libsnappy.so
Now after I restart the hadoop and tried my test MR job again, this time, it didn't give me
the originally error, but a new error like this:
Error: java.lang.UnsatisfiedLinkError: org/apache/hadoop/io/compress/snappy/SnappyCompressor.compressBytesDirect()I
at org.apache.hadoop.io.compress.snappy.SnappyCompressor.compress(SnappyCompressor.java:229)
at org.apache.hadoop.io.compress.BlockCompressorStream.compress(BlockCompressorStream.java:141)
at org.apache.hadoop.io.compress.BlockCompressorStream.finish(BlockCompressorStream.java:135)
at org.apache.hadoop.mapred.IFile$Writer.close(IFile.java:135)	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1450)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.access$1800(MapTask.java:852)	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer$SpillThread.run(MapTask.java:1343)
I wrote a test problem, like hadoop did load the library:
it works fine in my test program.
I don't know why at runtime, the Class SnappyCompressor.compressByteDirect() gave back that
kind of error. From the source code, it looks like a native c program from here
Any idea why this is happening on my environment?
View raw message