hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael Noll (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-2009) configure script for compiling hadoop native doesn't set lzo lib name correctly
Date Thu, 03 Nov 2011 15:03:32 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-2009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13143228#comment-13143228
] 

Michael Noll commented on HADOOP-2009:
--------------------------------------

FYI: The reason might be a problem with ld and how it is configured to work by default.  We
have run into basically the exact same error when trying to build the hadoop-lzo libraries
from [1].

I described the problem (for hadoop-lzo) in length here:
https://github.com/kevinweil/hadoop-lzo/issues/33

In a nutshell, the problem is that some systems run ld with a default setting of {{"\--as-needed"}}
whereas others have {{"\--no-as-needed"}}.  The build will only work if you run ld with the
latter, i.e. {{"\--no-as-needed"}}.  We run into this error when we tried to build hadoop-lzo
on Ubuntu 11.10, which is the first Ubuntu version where the default behavior of ld has been
changed to {{\--as-needed}}.

Now the reason why this has an effect is the way that the Hadoop LZO library is being searched
for in {{src/native/configure}}.  It is using a dummy C script and compiles it by running
gcc with "-llzo2":

{code}
// from src/native/configure:
echo 'int main(int argc, char **argv){return 0;}' > conftest.c
{code}

However, the dummy C script does not actually use lzo2 directly, so whether the lzo2 library
will be linked or not depends now on whether ld is being run with {{"\--as-needed"}} or with
{{"\--no-as-needed"}}.

I provided a patch [2] for hadoop-lzo that fixes this problem by setting {{LDFLAGS}} (env
var) in ant's {{build.xml}}.  The same fix solves this build error for Hadoop 0.20.203.0,
too:

{code}
    <exec dir="${build.native}" executable="sh" failonerror="true">
          <env key="OS_NAME" value="${os.name}"/>
          <env key="OS_ARCH" value="${os.arch}"/>
          <env key="JVM_DATA_MODEL" value="${sun.arch.data.model}"/>
          <env key="HADOOP_NATIVE_SRCDIR" value="${native.src.dir}"/>
          <env key="LDFLAGS" value="-Wl,--no-as-needed"/>     <== add this line
          <arg line="${native.src.dir}/configure"/>
    </exec>
{code}

If this seems like a reasonable approach (I'm not a C expert) I can provide a similar patch
for Hadoop 0.20.20x as well.

[1] https://github.com/kevinweil/hadoop-lzo
[2] https://github.com/kevinweil/hadoop-lzo/pull/34
                
> configure script for compiling hadoop native doesn't set lzo lib name correctly
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-2009
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2009
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.13.1
>         Environment: Fedora, amd64
>            Reporter: Joydeep Sen Sarma
>            Assignee: Arun C Murthy
>
> Looks like this was already reported (but not resolved on the the list): http://tinyurl.com/2rwu6x
> I would like to compile libhadoop on amd64/Fedora and everything seems kosher until I
hit this compile error:
>      [exec] /home/jssarma/fbprojects/hadoop-0.13.1/src/native/src/org/apache/hadoop/io/compress/lzo/LzoCompressor.c:116:
error: syntax error before ',' token
> the line in question is:
>     // Load liblzo2.so                                                              
                     
>     liblzo2 = dlopen(HADOOP_LZO_LIBRARY, RTLD_LAZY | RTLD_GLOBAL);
> seems like this is being set by:
> configure:#define HADOOP_LZO_LIBRARY ${ac_cv_libname_lzo2}
> I tried executing the relevant part of configure by hand:
>   if test -z "`${CC} -o conftest conftest.c -llzo2 2>&1`"; then
>         if test ! -z "`which objdump`"; then
>       ac_cv_libname_lzo2="`objdump -p conftest | grep NEEDED | grep lzo2 | sed 's/\W*NEEDED\W*\(.*\)\W*$/\
> \"\1\"/'`"
> This is not working on my system, since:
> > objdump -p conftest | grep NEEDED                         
>   NEEDED      libc.so.6
> So that would explain the compile error. Editing the configure script manually for now
works.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message