hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Nauroth (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-11127) Improve versioning and compatibility support in native library for downstream hadoop-common users.
Date Thu, 08 Oct 2015 21:04:27 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-11127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14949388#comment-14949388
] 

Chris Nauroth commented on HADOOP-11127:
----------------------------------------

[~alanburlison], thank you again for picking this up.  The proposal essentially jives with
the HADOOP-11064.003.patch attached to this JIRA, which is something Colin had coded a while
ago.

I agree with Colin that option #3 is possible.  I might not have made this clear in my earlier
notes, but the proposal is that hadoop-common.jar would contain a compatible native build
for every supported target platform, not just one.  It would contain a liibhadoop.so, hadoop.dll,
winutils.exe, and now Solaris artifacts too thanks to your efforts.  At runtime, hadoop-common.jar
can detect the platform and bind to the corresponding native artifact.  This has a lot of
benefits, because linking to a correct version would then be achieved completely by the Maven
dependency resolution process.  There wouldn't be any external dependencies.  This might even
completely eliminate a common category of problems for end users on Windows: how to track
down a good hadoop.dll and winutils.exe and configure Hadoop to find them.

Steve mentioned snappy-java as an example of a project that already demonstrates that this
is possible.  Here is a link.

https://github.com/xerial/snappy-java

Additionally, hadoop-lzo recently added similar support for its native components.

https://github.com/twitter/hadoop-lzo/pull/81

Option #3 also is admittedly the most difficult to implement.  The coding isn't the difficult
part.  The difficult part is the build and release process, which would look much different
from the current Apache Hadoop release process.

Unfortunately, I think versioning with a composite name introduces some new problems.  Steve
already articulated this very well.  Also, in the examples in your proposal, it uses X.Y versions,
i.e. libhadoop-2.7.so, without the .Z part.  Were you thinking of expanding this to include
the .Z part, i.e. libhadoop-2.7.1.so, or were you thinking that the internal interfaces are
still frozen within an X.Y version?

> Improve versioning and compatibility support in native library for downstream hadoop-common
users.
> --------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-11127
>                 URL: https://issues.apache.org/jira/browse/HADOOP-11127
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: native
>            Reporter: Chris Nauroth
>            Assignee: Alan Burlison
>         Attachments: HADOOP-11064.003.patch, proposal.txt
>
>
> There is no compatibility policy enforced on the JNI function signatures implemented
in the native library.  This library typically is deployed to all nodes in a cluster, built
from a specific source code version.  However, downstream applications that want to run in
that cluster might choose to bundle a hadoop-common jar at a different version.  Since there
is no compatibility policy, this can cause link errors at runtime when the native function
signatures expected by hadoop-common.jar do not exist in libhadoop.so/hadoop.dll.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message