hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Nauroth (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-11064) UnsatisifedLinkError with hadoop 2.4 JARs on hadoop-2.6 due NativeCRC32 method changes
Date Thu, 11 Sep 2014 18:08:35 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-11064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14130426#comment-14130426

Chris Nauroth commented on HADOOP-11064:

After further discussion, the following is a summary of potential solutions to this problem:
1. Freeze the libhadoop.so API forever.
2. Library versioning plus maintaining a library on the servers for each supported release.
3. Bundle the .dll or .so file inside a jar somehow so that YARN / Slider can distribute it.

#1. Advantages:
* ???

#1. Disadvantages:
  * We're unable to improve libhadoop.so in the future.
  * There will be puzzling interactions when mixing and matching versions.  "New" bugs in
libhadoop.so will show up with old hadoop releases, causing confusion in bug trackers.
  * We don't have any way of enforcing C API stability.  Jenkins doesn't check for it, most
Java programmers don't know how to achieve it.
  * There is still no ability for applications using new Hadoop versions to make use of old
libhadoop.so versions, unless we adopt an even worse compatibility policy that nothing new
can be added to libhadoop.so.
  * Given all of the above, this option seems to be off the table.

#2. Advantages:
  * Simple to implement.
  * There's already a patch that implements it.
  * We want libhadoop.so library versioning anyway, even if we later adopt another solution
in addition to this

#2. Disadvantages:
  * Admins using Slider / YARN will need to ensure that the appropriate versions of libhadoop
are present on the server.

#3. Advantages:
  * "Cleanest" solution, since it allows us to reuse YARN's existing distribution mechanisms.

#3. Disadvantages:
  * There are technical challenges to bundling a library in a jar that we haven't yet tackled.

Short-term, there is now agreement that the scope of this jira is to restore the 2 function
signatures that were removed to unblock downstream projects.

Colin plans to arrange a conference call for further discussion next week.  The details will
be posted here for anyone who wants to join, and we'll also call wider attention to this issue
on the mailing lists.

> UnsatisifedLinkError with hadoop 2.4 JARs on hadoop-2.6 due NativeCRC32 method changes
> --------------------------------------------------------------------------------------
>                 Key: HADOOP-11064
>                 URL: https://issues.apache.org/jira/browse/HADOOP-11064
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: native
>    Affects Versions: 2.6.0
>         Environment: Hadoop 2.6 cluster, trying to run code containing hadoop 2.4 JARs
>            Reporter: Steve Loughran
>            Assignee: Colin Patrick McCabe
>            Priority: Blocker
>         Attachments: HADOOP-11064.001.patch, HADOOP-11064.002.patch, HADOOP-11064.003.patch
> The private native method names and signatures in {{NativeCrc32}} were changed in HDFS-6561
... as a result hadoop-common-2.4 JARs get unsatisifed link errors when they try to perform
> This essentially stops Hadoop 2.4 applications running on Hadoop 2.6 unless rebuilt and
repackaged with the hadoop- 2.6 JARs

This message was sent by Atlassian JIRA

View raw message