spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kevin Markey <>
Subject Re: Spark 0.9.1 release
Date Tue, 25 Mar 2014 17:19:33 GMT

A correct shading of ASM should only affect Spark code unless someone is 
relying on ASM 4.0 in unrelated project code, in which case they can add 
org.ow2.asm:asm:4.x as a dependency.

Our short term solution has been to repackage other libraries with a 3.2 
dependency or to exclude ASM when our use of a dependent library really 
doesn't need it.  As you probably know, the real problem arises in 
ClassVisitor, which is an Interface in 3.x and before, but in 4.x it is 
an abstract class that takes a version constant as its constructor.  The 
ASM folks of course had our best interests in mind when they did this, 
attempting to deal with the Java-version dependent  changes from one ASM 
release to the next.  Unfortunately, they didn't change the names or 
locations of their classes and interfaces, which would have helped.

In our particular case, the only library from which we couldn't exclude 
ASM was 
org.glassfish.jersey.containers:jersey-container-servlet:jar:2.5.1. I 
added a new module to our project, including some dummy source code, 
because we needed the library to be self contained, made the servlet -- 
minus some unrelated transitive dependencies -- the only module 
dependency, then used the Maven shade plugin to relocate 
"org.objectweb.asm" to an arbitrary target.  We added the new shaded 
module as a new project dependency, plus the unrelated transitive 
dependencies excluded above.   This solved the problem. At least until 
we added WADL to the project.  Then we needed to deal with it on its own 

As you can see, we left Spark alone in all its ASM 4.0 glory.  Why? 
Spark is more volatile than the other libraries.  Also, the way in which 
we needed to deploy Spark and other resources on our (Yarn) clusters 
suggested that it would be easier to shade the other libraries.  I 
wanted to avoid having to install a locally patched Spark library into 
our build, updating the cluster and individual developers whenever 
there's a new patch.  Individual developers such as me who are testing 
the impact of patches can handle it, but the main build goes to Maven 
Central via our corporate Artifactory mirror.

If suddenly we had a Spark 0.9.1 with a shaded ASM, it would have no 
negative impact on us.  Only a positive impact.

I just wish that all users of ASM would read FAQ entry 15!!!


On 03/24/2014 06:30 PM, Tathagata Das wrote:
> Hello Kevin,
> A fix for SPARK-782 would definitely simplify building against Spark.
> However, its possible that a fix for this issue in 0.9.1 will break
> the builds (that reference spark) of existing 0.9 users, either due to
> a change in the ASM version, or for being incompatible with their
> current workarounds for this issue. That is not a good idea for a
> maintenance release, especially when 1.0 is not too far away.
> Can you (and others) elaborate more on the current workarounds that
> you have for this issue? Its best to understand all the implications
> of this fix.
> Note that in branch 0.9, it is not fixed, neither in SBT nor in Maven.
> TD
> On Mon, Mar 24, 2014 at 4:38 PM, Kevin Markey <> wrote:
>> Is there any way that [SPARK-782] (Shade ASM) can be included?  I see that
>> it is not currently backported to 0.9.  But there is no single issue that
>> has caused us more grief as we integrate spark-core with other project
>> dependencies.  There are way too many libraries out there in addition to
>> Spark 0.9 and before that are not well-behaved (ASM FAQ recommends shading),
>> including some Hive and Hadoop libraries and a number of servlet libraries.
>> We can't control those, but if Spark were well behaved in this regard, it
>> would help.  Even for a maintenance release, and even if 1.0 is only 6 weeks
>> away!
>> (For those not following 782, according to Jira comments, the SBT build
>> shades it, but it is the Maven build that ends up in Maven Central.)
>> Thanks
>> Kevin Markey
>> On 03/19/2014 06:07 PM, Tathagata Das wrote:
>>>    Hello everyone,
>>> Since the release of Spark 0.9, we have received a number of important bug
>>> fixes and we would like to make a bug-fix release of Spark 0.9.1. We are
>>> going to cut a release candidate soon and we would love it if people test
>>> it out. We have backported several bug fixes into the 0.9 and updated JIRA
>>> accordingly<>.
>>> Please let me know if there are fixes that were not backported but you
>>> would like to see them in 0.9.1.
>>> Thanks!
>>> TD

View raw message