hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andy Isaacson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-8901) GZip and Snappy support may not work without unversioned libraries
Date Mon, 22 Oct 2012 22:42:12 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-8901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13481890#comment-13481890

Andy Isaacson commented on HADOOP-8901:

bq. I'll test your suggestion on Linux to see if it has the needed behavior

I tested with
--- a/hadoop-common-project/hadoop-common/src/CMakeLists.txt
+++ b/hadoop-common-project/hadoop-common/src/CMakeLists.txt
@@ -55,7 +55,7 @@ if (NOT GENERATED_JAVAH)
     MESSAGE(FATAL_ERROR "You must set the cmake variable GENERATED_JAVAH")
 find_package(JNI REQUIRED)
-find_package(ZLIB REQUIRED)
+find_package(ZLIB 1 REQUIRED)
 set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -g -Wall -O2")

and found that the resulting value for {{HADOOP_ZLIB_LIBRARY}} remained {{"libz.so"}}.  This
doesn't work on standard Linux installs because {{dlopen}} uses the string it is passed as
an exact filename match, and without the {{-dev}} packages, Linux installations do not have
{{libfoo.so}} symlinks installed.

Colin's fix which was committed on this Jira addresses the problem by ensuring that {{HADOOP_ZLIB_LIBRARY}}
expands to a string that specifies the library ABI version, {{"libz.so.1"}}.

It seems like you've actually got the same problem on BSD, but the hardcoded "1" breaks because
BSD considers zlib to have a larger ABI revision.  Possibly this is related to an issue in
the upstream Linux system where the zlib ABI number is not properly being incremented when
the ABI changes -- certainly BSD has been much more careful / aware of such issues over the

I think the right fix is to figure out how to have CMake determine the correct ABI revision
number at build time, no?
> GZip and Snappy support may not work without unversioned libraries
> ------------------------------------------------------------------
>                 Key: HADOOP-8901
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8901
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: native
>    Affects Versions: 2.0.3-alpha
>            Reporter: Colin Patrick McCabe
>            Assignee: Colin Patrick McCabe
>            Priority: Minor
>             Fix For: 3.0.0, 2.0.3-alpha
>         Attachments: HADOOP-8901.001.patch, HADOOP-8901.002.patch, HADOOP-8901.003.patch
> Currently, we use {{dlopen}} to open {{libz.so}} and {{libsnappy.so}}, to get Gzip and
Snappy support, respectively.
> However, this is not correct; we should be dlopening {{libsnappy.so.1}} instead.  The
versionless form of the shared library is not commonly installed except by development packages.
 Also, we may run into subtle compatibility problems if a new version of libsnappy comes out.
> Thanks to Brandon Vargo for reporting this bug.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message