hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From java8964 java8964 <java8...@hotmail.com>
Subject RE: enable snappy on hadoop 1.1.1
Date Fri, 04 Oct 2013 21:07:28 GMT
I kind of read the hadoop 1.1.1 source code for this, it is very strange for me now.
>From the error, it looks like runtime JVM cannot find the native method of org/apache/hadoop/io/compress/snappy/SnappyCompressor.compressBytesDirect()I,
that my guess from the error message, but from the log, it looks like all the native library,
include native-hadoop and native snappy are both loaded, as shown in the failed task log:
2013-10-04 16:33:21,635 INFO org.apache.hadoop.util.NativeCodeLoader: Loaded the native-hadoop
library2013-10-04 16:33:22,006 INFO org.apache.hadoop.util.ProcessTree: setsid exited with
exit code 02013-10-04 16:33:22,020 INFO org.apache.hadoop.mapred.Task:  Using ResourceCalculatorPlugin
: org.apache.hadoop.util.LinuxResourceCalculatorPlugin@282528252013-10-04 16:33:22,111 INFO
org.apache.hadoop.mapred.MapTask: numReduceTasks: 12013-10-04 16:33:22,116 INFO org.apache.hadoop.mapred.MapTask:
io.sort.mb = 2562013-10-04 16:33:22,168 INFO org.apache.hadoop.mapred.MapTask: data buffer
= 204010960/2550136962013-10-04 16:33:22,168 INFO org.apache.hadoop.mapred.MapTask: record
buffer = 671088/8388602013-10-04 16:33:22,342 WARN org.apache.hadoop.io.compress.snappy.LoadSnappy:
Snappy native library is available2013-10-04 16:33:22,342 INFO org.apache.hadoop.io.compress.snappy.LoadSnappy:
Snappy native library loaded2013-10-04 16:33:44,054 INFO org.apache.hadoop.mapred.MapTask:
Starting flush of map output2013-10-04 16:33:44,872 WARN org.apache.hadoop.io.compress.snappy.SnappyCompressor:
java.lang.UnsatisfiedLinkError: org/apache/hadoop/io/compress/snappy/SnappyCompressor.initIDs()V2013-10-04
16:33:44,872 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor2013-10-04
16:33:44,928 INFO org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs' truncater
with mapRetainSize=-1 and reduceRetainSize=-12013-10-04 16:33:44,951 INFO org.apache.hadoop.io.nativeio.NativeIO:
Initialized cache for UID to User mapping with a cache timeout of 14400 seconds.2013-10-04
16:33:44,951 INFO org.apache.hadoop.io.nativeio.NativeIO: Got UserName yzhang for UID 1000
from the native implementation2013-10-04 16:33:44,952 FATAL org.apache.hadoop.mapred.Child:
Error running child : java.lang.UnsatisfiedLinkError: org/apache/hadoop/io/compress/snappy/SnappyCompressor.compressBytesDirect()I
       at org.apache.hadoop.io.compress.snappy.SnappyCompressor.compress(SnappyCompressor.java:229)
       at org.apache.hadoop.io.compress.BlockCompressorStream.compress(BlockCompressorStream.java:141)
       at org.apache.hadoop.io.compress.BlockCompressorStream.finish(BlockCompressorStream.java:135)
       at org.apache.hadoop.mapred.IFile$Writer.close(IFile.java:135)        at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1450)
       at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1297)      
 at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:371)
       at org.apache.hadoop.mapred.Child$4.run(Child.java:255)        at java.security.AccessController.doPrivileged(AccessController.java:310)
       at javax.security.auth.Subject.doAs(Subject.java:573)        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
       at org.apache.hadoop.mapred.Child.main(Child.java:249)
So is there any way I can check if $HADOOP_HOME/lib/native/Linux-amd64-64/libhadoop.so contains
the native method as expected above?This hadoop 1.1.0 is not compiled by me, but coming from
IBM biginsight 2.1 as we are evaluating it. I will create a ticket for them, but is this kind
of strange, as everything loaded shown in the log, but complains later about a native method?
Any reason can cause this?
Yong
From: java8964@hotmail.com
To: user@hadoop.apache.org
Subject: enable snappy on hadoop 1.1.1
Date: Fri, 4 Oct 2013 15:44:34 -0400




Hi, 
I am using hadoop 1.1.1. I want to test to see the snappy compression with hadoop, but I have
some problems to make it work on my Linux environment.
I am using opensuse 12.3 x86_64. 
First, when I tried to enable snappy in hadoop 1.1.1 by:
            conf.setBoolean("mapred.compress.map.output", true);            conf.set("mapred.output.compression.type",
"RECORD");            conf.set("mapred.map.output.compression.codec", "org.apache.hadoop.io.compress.SnappyCodec");
I got the following error in my test MR job:
Exception in thread "main" java.lang.RuntimeException: native snappy library not available
So I download the snappy 1.1.0 from https://code.google.com/p/snappy/, compile it and install
it successfully under /opt/snappy-1.1.0, and then I link the /opt/snappy-1.1.0/lib64/libsnappy.so
to /user/lib64/libsnappy.so
Now after I restart the hadoop and tried my test MR job again, this time, it didn't give me
the originally error, but a new error like this:
Error: java.lang.UnsatisfiedLinkError: org/apache/hadoop/io/compress/snappy/SnappyCompressor.compressBytesDirect()I
at org.apache.hadoop.io.compress.snappy.SnappyCompressor.compress(SnappyCompressor.java:229)
at org.apache.hadoop.io.compress.BlockCompressorStream.compress(BlockCompressorStream.java:141)
at org.apache.hadoop.io.compress.BlockCompressorStream.finish(BlockCompressorStream.java:135)
at org.apache.hadoop.mapred.IFile$Writer.close(IFile.java:135)	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1450)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.access$1800(MapTask.java:852)	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer$SpillThread.run(MapTask.java:1343)
I wrote a test problem, like hadoop did load the library:
System.loadlibrary("snappy")
it works fine in my test program.
I don't know why at runtime, the Class SnappyCompressor.compressByteDirect() gave back that
kind of error. From the source code, it looks like a native c program from here
https://code.google.com/p/hadoop-snappy/source/browse/trunk/src/main/java/org/apache/hadoop/io/compress/snappy/SnappyCompressor.java
Any idea why this is happening on my environment?
Thanks
Yong 		 	   		   		 	   		  
Mime
View raw message