hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andy Isaacson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-8806) libhadoop.so: dlopen should be better at locating libsnappy.so, etc.
Date Sat, 15 Sep 2012 02:41:07 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-8806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13456291#comment-13456291

Andy Isaacson commented on HADOOP-8806:

I ran a standalone test of {{DT_RPATH $ORIGIN}}.  There's good news and bad news which is
probably ok.

On the plus side, setting DT_RPATH=$ORIGIN on libhadoop.so does allow it to find libsnappy.so
in the same directory.  This is good.

On the downside, setting DT_RPATH=$ORIGIN on libhadoop.so pollutes the search path for later
dlopens in the main executable.  In my test,
    -> dlopens /path/to/libbar.so with DT_RPATH=$ORIGIN
          -> dlopens libfoo.so
    -> dlopens libfoo.so
Since main.c is supposed to be unaffected by the behavior of libbar, the final dlopen should
fail since main's search path does not include libfoo.

Unfortunately the final dlopen succeeds.  This means that the libfoo opened by libbar, while
loaded, is available for open by main.

Modifying the test so that libbar.so dlclose()s libfoo before returning fixes the problem;
the final open of libfoo fails again.  So it's not a problem of libbar's DT_RPATH polluting
the main executable's search path, but rather, a single table of currently open objects.

What does this mean?  Suppose we have a libsnappy.1.0.14 in $ORIGIN, and the system has a
libsnappy.1.0.20 in /usr/lib.  A program which uses libhadoop will get 1.0.14 if libhadoop
is opened before libsnappy, and 1.0.20 if libsnappy is opened before libhadoop.  This kinda
sucks. But, it's definitely a corner case, so maybe it's OK.

If we used RUNPATH rather than RPATH, the user could work around the above problem by setting
LD_LIBRARY_PATH.  Since it's probably safer from a build perspective to use RPATH -- I couldn't
even figure out how to get ld-2.22 to set RUNPATH -- then LD_LIBRARY_PATH will not work. 
> libhadoop.so: dlopen should be better at locating libsnappy.so, etc.
> --------------------------------------------------------------------
>                 Key: HADOOP-8806
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8806
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Colin Patrick McCabe
>            Assignee: Colin Patrick McCabe
>            Priority: Minor
>         Attachments: HADOOP-8806.003.patch
> libhadoop calls {{dlopen}} to load {{libsnappy.so}} and {{libz.so}}.  These libraries
can be bundled in the {{$HADOOP_ROOT/lib/native}} directory.  For example, the {{-Dbundle.snappy}}
build option copies {{libsnappy.so}} to this directory.  However, snappy can't be loaded from
this directory unless {{LD_LIBRARY_PATH}} is set to include this directory.
> Can we make this configuration "just work" without needing to rely on {{LD_LIBRARY_PATH}}?

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message