hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Colin Patrick McCabe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-11127) Improve versioning and compatibility support in native library for downstream hadoop-common users.
Date Mon, 21 Dec 2015 19:17:47 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-11127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15066916#comment-15066916

Colin Patrick McCabe commented on HADOOP-11127:

Thanks for thinking about this, [~cnauroth].

I'm concerned that if we implement fallback logic like you described, people will experience
weird behaviors and heisenbugs when the wrong libhadoop version is installed.  I think most
admins would rather see a simple and clear warning message that libhadoop-2.8.3 is not installed,
than see weird behaviors like one native function working and another not.  Not all changes
in the native library general clear error messages either... something like changing an uint32_t
to a uint64_t could easily lead to crashes or buffer overflows rather than a "dlsym failed"
error message.

I think we should check only for the specific version of the library that ought to be installed,
not for any other.  After all, we already have a fallback mode where we run without JNI. 
If installing the correct libhadoop versions is too much of a burden, admins can run in that
mode.  We just don't have the testing bandwidth to make sure an arbitrary number of fallback
configurations work as advertised (or even fail cleanly as advertised.)

I agree that we need to handle Windows, and that we should consult with BigTop-- those are
good points.

> Improve versioning and compatibility support in native library for downstream hadoop-common
> --------------------------------------------------------------------------------------------------
>                 Key: HADOOP-11127
>                 URL: https://issues.apache.org/jira/browse/HADOOP-11127
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: native
>            Reporter: Chris Nauroth
>            Assignee: Alan Burlison
>         Attachments: HADOOP-11064.003.patch, proposal.01.txt
> There is no compatibility policy enforced on the JNI function signatures implemented
in the native library.  This library typically is deployed to all nodes in a cluster, built
from a specific source code version.  However, downstream applications that want to run in
that cluster might choose to bundle a hadoop-common jar at a different version.  Since there
is no compatibility policy, this can cause link errors at runtime when the native function
signatures expected by hadoop-common.jar do not exist in libhadoop.so/hadoop.dll.

This message was sent by Atlassian JIRA

View raw message