hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Colin Patrick McCabe (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HADOOP-8756) libsnappy loader issues
Date Thu, 06 Sep 2012 22:28:09 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-8756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Colin Patrick McCabe updated HADOOP-8756:
-----------------------------------------

    Attachment: HADOOP-8756.002.patch

* remove HADOOP_RUNAS_HOME (runAs has been removed)

* remove SnappyCodec.java.  Instead, load the native code from SnappyCompressor.java and SnappyDecompressor.java.
 This is similar to the way the zlib stuff works now.

* If we try to instantiate SnappyCodec, but snappy is not loaded, throw an exception.  Formerly,
we might get undefined behavior like segfaults in this case.

* Be more helpful about why snappy could not be loaded: was it because the build was compiled
without snappy?  Or some other reason found in the exception thrown from {{SnappyDecompressor#initIDs}}
or  {{SnappyCompressor#initIDs}}?

* We don't need to call {{System.loadLibrary("snappy")}}.  It's irrelevant because libsnappy
is not a JNI library.  Again, zlib shows how this should be done: just use {{dlopen}}, not
{{System.loadLibrary}}.  Note that calliing {{System.loadLibrary}} does *not* make the symbols
visible to {{libhadoop.so}} because the JVM does not use {{RTLD_GLOBAL}}. 
                
> libsnappy loader issues
> -----------------------
>
>                 Key: HADOOP-8756
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8756
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: native
>    Affects Versions: 2.2.0-alpha
>            Reporter: Colin Patrick McCabe
>            Assignee: Colin Patrick McCabe
>            Priority: Minor
>         Attachments: HADOOP-8756.002.patch
>
>
> We use {{System.loadLibrary("snappy")}} from the Java side.  However in libhadoop, we
use {{dlopen}} to open libsnappy.so dynamically.  System.loadLibrary uses {{java.library.path}}
to resolve libraries, and {{dlopen}} uses {{LD_LIBRARY_PATH}} and the system paths to resolve
libraries.  Because of this, the two library loading functions can be at odds.
> We should fix this so we only load the library once, preferably using the standard Java
{{java.library.path}}.
> We should also log the search path(s) we use for {{libsnappy.so}} when loading fails,
so that it's easier to diagnose configuration issues.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message