spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Patrick Wendell (JIRA)" <>
Subject [jira] [Resolved] (SPARK-1789) Multiple versions of Netty dependencies cause FlumeStreamSuite failure
Date Sun, 11 May 2014 03:51:14 GMT


Patrick Wendell resolved SPARK-1789.

       Resolution: Fixed
    Fix Version/s: 1.0.0

Issue resolved by pull request 723

> Multiple versions of Netty dependencies cause FlumeStreamSuite failure
> ----------------------------------------------------------------------
>                 Key: SPARK-1789
>                 URL:
>             Project: Spark
>          Issue Type: Bug
>          Components: Build
>    Affects Versions: 0.9.1
>            Reporter: Sean Owen
>              Labels: flume, netty, test
>             Fix For: 1.0.0
> TL;DR is there is a bit of JAR hell trouble with Netty, that can be mostly resolved and
will resolve a test failure.
> I hit the error described at
while running FlumeStreamingSuite, and have for a short while (is it just me?)
> velvia notes:
> "I have found a workaround.  If you add akka 2.2.4 to your dependencies, then everything
works, probably because akka 2.2.4 brings in newer version of Jetty." 
> There are at least 3 versions of Netty in play in the build:
> - the new Flume 1.4.0 dependency brings in io.netty:netty:3.4.0.Final, and that is the
immediate problem
> - the custom version of akka 2.2.3 depends on io.netty:netty:3.6.6.
> - but, Spark Core directly uses io.netty:netty-all:4.0.17.Final
> The POMs try to exclude other versions of netty, but are excluding org.jboss.netty:netty,
when in fact older versions of io.netty:netty (not netty-all) are also an issue.
> The org.jboss.netty:netty excludes are largely unnecessary. I replaced many of them with
io.netty:netty exclusions until everything agreed on io.netty:netty-all:4.0.17.Final.
> But this didn't work, since Akka 2.2.3 doesn't work with Netty 4.x. Down-grading to 3.6.6.Final
across the board made some Spark code not compile.
> If the build *keeps* io.netty:netty:3.6.6.Final as well, everything seems to work. Part
of the reason seems to be that Netty 3.x used the old `org.jboss.netty` packages. This is
less than ideal, but is no worse than the current situation. 
> So this PR resolves the issue and improves the JAR hell, even if it leaves the existing
theoretical Netty 3-vs-4 conflict:
> - Remove org.jboss.netty excludes where possible, for clarity; they're not needed except
with Hadoop artifacts
> - Add io.netty:netty excludes where needed -- except, let akka keep its io.netty:netty
> - Change a bit of test code that actually depended on Netty 3.x, to use 4.x equivalent
> - Update SBT build accordingly
> A better change would be to update Akka far enough such that it agrees on Netty 4.x,
but I don't know if that's feasible.

This message was sent by Atlassian JIRA

View raw message