Mailing-List: contact dev-help@storm.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@storm.incubator.apache.org
Received-SPF: pass (nike.apache.org: domain of m.ben.franklin@gmail.com
 designates 209.85.216.54 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <CF1A7F77.24C24%evans@yahoo-inc.com>
References: 
 <CAKgXx6LWetozLvv6RCHYamD0gXwNvLMi9-NnhM4ghQn=C5+nfA@mail.gmail.com>
	<62C4B740-3DF3-4D48-8664-6BFBE86388C7@gmail.com>
	<CA+ZeGPB7mh5_yLieB+VeA879A=MqP3FH4EgAxhfm-DcDPz07ow@mail.gmail.com>
	<CAJgRgoushtddjUdpqVMT_o0PF2rAKg-oLNGTGz7MXiEiX3v3dA@mail.gmail.com>
	<CAKgXx6Kz7dKg95mCskn9GQnsULR10RUZDh8WPd=755K8WGMgwg@mail.gmail.com>
	<A732B22E-1D47-4512-B876-5FA7A28D1E8F@gmail.com>
	<CAD9ohx-ovgzZ1CnzZKOi+o1TkTZ+MyhddN_ekY-ydOFRyWTAWQ@mail.gmail.com>
	<09772E04-68A1-42E8-821D-818E919D4759@gmail.com>
	<CAOW7FxYxF=7JpFT_GFnnJobOR7+U7KjUwTJkDZO+x_ONuTX8hw@mail.gmail.com>
	<CF1A7F77.24C24%evans@yahoo-inc.com>
Date: Fri, 7 Feb 2014 15:36:50 -0500
Message-ID: 
 <CAENpzYWicKypBWmKiegu6K+vd8d=YgANwVuG7aZ0FD7jEEdhFA@mail.gmail.com>
Subject: Re: http-client version conflict
From: Matt Franklin <m.ben.franklin@gmail.com>
To: "dev@storm.incubator.apache.org" <dev@storm.incubator.apache.org>
Content-Type: multipart/alternative; boundary=001a113aadc2144edc04f1d6f3bb

--001a113aadc2144edc04f1d6f3bb
Content-Type: text/plain; charset=ISO-8859-1

On Fri, Feb 7, 2014 at 1:21 PM, Bobby Evans <evans@yahoo-inc.com> wrote:

> I am +1 for relocating storm's dependencies, but I also think that we want
> to split some parts out too.  There are a number of things on the worker
> classpath that the worker just does not need.  Jetty for example.  I think
> in these cases it seems simpler to have the worker process/client process
> and it's dependencies separated out from other storm system daemons.  And
> then only look at relocating the dependencies the client/worker uses.
>

+1.  This makes the most sense to me.


>
> --Bobby
>
> From: "Cody A. Ray" <cody.a.ray@gmail.com<mailto:cody.a.ray@gmail.com>>
> Reply-To: "user@storm.incubator.apache.org<mailto:
> user@storm.incubator.apache.org>" <user@storm.incubator.apache.org<mailto:
> user@storm.incubator.apache.org>>
> Date: Thursday, February 6, 2014 at 9:09 PM
> To: "user@storm.incubator.apache.org<mailto:
> user@storm.incubator.apache.org>" <user@storm.incubator.apache.org<mailto:
> user@storm.incubator.apache.org>>
> Cc: "dev@storm.incubator.apache.org<mailto:dev@storm.incubator.apache.org>"
> <dev@storm.incubator.apache.org<mailto:dev@storm.incubator.apache.org>>
> Subject: Re: http-client version conflict
>
> +1 for relocating storm's dependencies. I just ran into this a couple
> weeks ago and have had to side-shelf the project until I find a
> work-around. Will check-out the shade plugin. Thanks for the rec!
>
> -Cody
>
>
> On Thu, Feb 6, 2014 at 9:07 PM, P. Taylor Goetz <ptgoetz@gmail.com<mailto:
> ptgoetz@gmail.com>> wrote:
> I tend to agree.
>
> The move to maven makes package relocation very easy, but we should be
> careful not to abuse it if that's a direction we decide to take.
>
> IMHO, users having to modify the contents of $STORM_HOME/lib is really bad
> and should only be done as a last resort. If someone asks for help with an
> issue after they've modified the contents of $STORM_HOM/lib, all bets are
> off.
>
> I also agree that we need to keep a close watch on our dependencies, and
> that some might warrant re-evaluation.
>
> I'd love to others opinions.
>
> - Taylor
>
> On Feb 6, 2014, at 9:28 PM, Adam Lewis <mail@adamlewis.com<mailto:
> mail@adamlewis.com>> wrote:
>
> My $0.02 on this subject:
>
> Without going down the path of class loader or OSGi mania and becoming a
> full container, I'd definitely be in favor of storm relocating its own
> dependencies.  In this way edge cases around things like reflection can be
> handled once within storm rather than burdening every topology builder with
> those details.  Part of the problem seems to be that storm makes extensive
> use (directly or transitively) of a lot of go-to utility libraries like
> guava, thrift, jodatime, json-simple, snakeyaml, commons-io, etc... I'm
> sure that leveraging these libraries allowed storm's development to proceed
> rapidly, but from a maturity perspective, it is problematic to impose these
> version choices on users.  And while I might want Storm to, say, try to
> track the latest Guava version, that same policy could be very problematic
> for others.
>
> If storm can relocate even some of its own dependencies, I think that
> would be a great help to me at least.  Longer term, I wonder how much of
> some of these libraries are really being used.  For example, is clj-time
> (and by extension joda-time) really needed? Or just a small fraction of the
> functionality in that library?  I can probably pitch in some of the effort
> required to do this, if this is the direction people want to go in.
>
>
>
>
> On Thu, Feb 6, 2014 at 8:44 PM, P. Taylor Goetz <ptgoetz@gmail.com<mailto:
> ptgoetz@gmail.com>> wrote:
> I'm glad the shader plugin worked for you.
>
> Updating dependencies can be tricky as it can easily introduce regressions.
>
> Ultimately we need to figure out the best solution to avoiding conflicts
> between user code (i.e. dependencies in topology jar files) and Storm's
> libraries.
>
> The classloader approach has been attempted, but IMO Storm's use of
> serialization complicates things significantly. Package relocation seems to
> be a relatively lightweight solution.
>
> If that's a direction we pursue, then it introduces the question of
> whether Storm should relocate its dependencies, or if that should be left
> up to the user (topology developer).
>
> Elastic Search has gone down the path of relocating some of their
> dependencies [1] (not necessarily an endorsement, just an observation).
>
> I've CC'd dev@ since this is all related to the infamous issue #115,
> which is now STORM-129 [2].
>
> - Taylor
>
> [1]
> https://github.com/elasticsearch/elasticsearch/blob/master/pom.xml#L474
> [2] https://issues.apache.org/jira/browse/STORM-129
>
>
>
>
>
> On Feb 6, 2014, at 7:25 PM, Vinay Pothnis <vinay.pothnis@gmail.com<mailto:
> vinay.pothnis@gmail.com>> wrote:
>
> Thank you all for replies! The shader-plugin solution seems to work for us.
>
> I wonder if we can create a JIRA ticket for storm to upgrade the
> http-client library as part of their next release.
>
> -Vinay
>
>
> On Thu, Feb 6, 2014 at 2:38 PM, Michael Rose <michael@fullcontact.com
> <mailto:michael@fullcontact.com>> wrote:
> We've done this with SLF4j and Guava as well without issues.
>
>
> Michael Rose (@Xorlev<https://twitter.com/xorlev>)
> Senior Platform Engineer, FullContact<http://www.fullcontact.com/>
> michael@fullcontact.com<mailto:michael@fullcontact.com>
>
>
> On Thu, Feb 6, 2014 at 3:03 PM, Mark Greene <mark@evertrue.com<mailto:
> mark@evertrue.com>> wrote:
> We had this problem as well. We modified our chef cookbook to just replace
> the older version with the newer one and storm didn't complain or have any
> other issues as a result.
>
>
> On Wed, Feb 5, 2014 at 10:31 AM, P. Taylor Goetz <ptgoetz@gmail.com
> <mailto:ptgoetz@gmail.com>> wrote:
> Your best bet is probably  to use the shade plugin to relocate the
> http-client package so it doesn't conflict with the version storm uses.
>
> Storm does this with the libtrhift dependency in storm-core:
>
>
> https://github.com/apache/incubator-storm/blob/master/storm-core/pom.xml#L220
>
> (You can ignore the clojure transformer in that config, unless you have
> non-AOT clojure code that uses the http-client library).
>
> More information on using the shade plugin to do package relocations can
> be found here:
>
>
> http://maven.apache.org/plugins/maven-shade-plugin/examples/class-relocation.html
>
> - Taylor
>
> On Feb 4, 2014, at 4:27 PM, Vinay Pothnis <vinay.pothnis@gmail.com<mailto:
> vinay.pothnis@gmail.com>> wrote:
>
> > Hello,
> >
> > I am using storm version 0.9.0.1.
> > My application depends on apache http-client version 4.3.2 - but storm
> depends on http-client version 4.1.1.
> >
> > What is the best way to override this dependency?
> >
> > Thanks
> > Vinay
>
>
>
>
>
>
>
>
>
>
> --
> Cody A. Ray, LEED AP
> codyaray@drexel.edu<mailto:codyaray@drexel.edu>
> 215.501.7891
>

--001a113aadc2144edc04f1d6f3bb--