hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Colin Patrick McCabe (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HADOOP-8756) libsnappy loader issues
Date Thu, 06 Sep 2012 22:28:09 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-8756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Colin Patrick McCabe updated HADOOP-8756:

    Attachment: HADOOP-8756.002.patch

* remove HADOOP_RUNAS_HOME (runAs has been removed)

* remove SnappyCodec.java.  Instead, load the native code from SnappyCompressor.java and SnappyDecompressor.java.
 This is similar to the way the zlib stuff works now.

* If we try to instantiate SnappyCodec, but snappy is not loaded, throw an exception.  Formerly,
we might get undefined behavior like segfaults in this case.

* Be more helpful about why snappy could not be loaded: was it because the build was compiled
without snappy?  Or some other reason found in the exception thrown from {{SnappyDecompressor#initIDs}}
or  {{SnappyCompressor#initIDs}}?

* We don't need to call {{System.loadLibrary("snappy")}}.  It's irrelevant because libsnappy
is not a JNI library.  Again, zlib shows how this should be done: just use {{dlopen}}, not
{{System.loadLibrary}}.  Note that calliing {{System.loadLibrary}} does *not* make the symbols
visible to {{libhadoop.so}} because the JVM does not use {{RTLD_GLOBAL}}. 
> libsnappy loader issues
> -----------------------
>                 Key: HADOOP-8756
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8756
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: native
>    Affects Versions: 2.2.0-alpha
>            Reporter: Colin Patrick McCabe
>            Assignee: Colin Patrick McCabe
>            Priority: Minor
>         Attachments: HADOOP-8756.002.patch
> We use {{System.loadLibrary("snappy")}} from the Java side.  However in libhadoop, we
use {{dlopen}} to open libsnappy.so dynamically.  System.loadLibrary uses {{java.library.path}}
to resolve libraries, and {{dlopen}} uses {{LD_LIBRARY_PATH}} and the system paths to resolve
libraries.  Because of this, the two library loading functions can be at odds.
> We should fix this so we only load the library once, preferably using the standard Java
> We should also log the search path(s) we use for {{libsnappy.so}} when loading fails,
so that it's easier to diagnose configuration issues.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message