hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andy Isaacson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-8901) GZip and Snappy support may not work without unversioned libraries
Date Mon, 22 Oct 2012 22:42:12 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-8901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13481890#comment-13481890
] 

Andy Isaacson commented on HADOOP-8901:
---------------------------------------

bq. I'll test your suggestion on Linux to see if it has the needed behavior

I tested with
{noformat}
--- a/hadoop-common-project/hadoop-common/src/CMakeLists.txt
+++ b/hadoop-common-project/hadoop-common/src/CMakeLists.txt
@@ -55,7 +55,7 @@ if (NOT GENERATED_JAVAH)
     MESSAGE(FATAL_ERROR "You must set the cmake variable GENERATED_JAVAH")
 endif (NOT GENERATED_JAVAH)
 find_package(JNI REQUIRED)
-find_package(ZLIB REQUIRED)
+find_package(ZLIB 1 REQUIRED)
 
 set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -g -Wall -O2")
 set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -D_REENTRANT -D_FILE_OFFSET_BITS=64")
{noformat}

and found that the resulting value for {{HADOOP_ZLIB_LIBRARY}} remained {{"libz.so"}}.  This
doesn't work on standard Linux installs because {{dlopen}} uses the string it is passed as
an exact filename match, and without the {{-dev}} packages, Linux installations do not have
{{libfoo.so}} symlinks installed.

Colin's fix which was committed on this Jira addresses the problem by ensuring that {{HADOOP_ZLIB_LIBRARY}}
expands to a string that specifies the library ABI version, {{"libz.so.1"}}.

It seems like you've actually got the same problem on BSD, but the hardcoded "1" breaks because
BSD considers zlib to have a larger ABI revision.  Possibly this is related to an issue in
the upstream Linux system where the zlib ABI number is not properly being incremented when
the ABI changes -- certainly BSD has been much more careful / aware of such issues over the
years.

I think the right fix is to figure out how to have CMake determine the correct ABI revision
number at build time, no?
                
> GZip and Snappy support may not work without unversioned libraries
> ------------------------------------------------------------------
>
>                 Key: HADOOP-8901
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8901
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: native
>    Affects Versions: 2.0.3-alpha
>            Reporter: Colin Patrick McCabe
>            Assignee: Colin Patrick McCabe
>            Priority: Minor
>             Fix For: 3.0.0, 2.0.3-alpha
>
>         Attachments: HADOOP-8901.001.patch, HADOOP-8901.002.patch, HADOOP-8901.003.patch
>
>
> Currently, we use {{dlopen}} to open {{libz.so}} and {{libsnappy.so}}, to get Gzip and
Snappy support, respectively.
> However, this is not correct; we should be dlopening {{libsnappy.so.1}} instead.  The
versionless form of the shared library is not commonly installed except by development packages.
 Also, we may run into subtle compatibility problems if a new version of libsnappy comes out.
> Thanks to Brandon Vargo for reporting this bug.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message