hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alan Burlison <Alan.Burli...@oracle.com>
Subject Re: DomainSocket issues on Solaris
Date Wed, 07 Oct 2015 16:35:39 GMT
On 06/10/2015 10:52, Steve Loughran wrote:

> HADOOP-11127, "Improve versioning and compatibility support in native
> library for downstream hadoop-common users." says "we need to do
> better here", which is probably some way of packaging native libs.

 From that JIRA:

> Colin Patrick McCabe added a comment - 18/Apr/15 00:48
> I was thinking we:
> 1. Add the Hadoop release version to libhadoop.so. It's very, very
> simple and solves a lot of problems here.
> 2. Remove libhadoop.so and libhdfs.so from the release tarball, since
> they are CPU and OS-specific and the tarballs are not
> 3. Schedule some follow-on work to include the native libraries
> inside jars, as Chris suggested. This will take longer but ultimately
> be the best solution.


> I just spotted one: HADOOP-10027.  A field was removed from the Java
> layer, which still could get referenced by an older version of the native
> layer.  A backwards-compatible version of that patch would preserve the
> old fields in the Java layer.

I've been thinking about this and I really don't think the strategy of 
trying to shim old methods and fields back in to Hadoop is the correct 
one.  The current Java-JNI interactions have been developed in an ad-hoc 
manner with no formal API definition and are explicitly Not-An-Interface 
and as a result no consideration has been given to cross-version 
stability. A compatibility shim approach is neither sustainable nor 
maintainable even on a single platform, and will severely compromise 
efforts to get Hadoop native components working on other platforms.

The approach suggested in HADOOP-11127 seems a much better way forward, 
in particular #2 (versioned libhadoop). As pointed out in the JIRA, #1 
(freeze libahdoop forever) is an obvious non-starter, and #3 (distribute 
libahadoop inside the JAR) is also a non-starter as it will not work 

I'm happy to work on HADOOP-10027 and make that a prerequisite for 
fixing the Solaris DomainSocket issues discussed in this thread. I 
believe it's not practical to provide a fix for DomainSocket on Solaris 
with a 'No JNI signature changes' restriction.

Does that sound acceptable? If so I can draft up a proposal for native 
library version and platform naming, library search locations etc.


Alan Burlison

View raw message