Mailing-List: contact dev-help@storm.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@storm.incubator.apache.org
Received-SPF: pass (nike.apache.org: domain of cody.a.ray@gmail.com designates
 209.85.216.172 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <09772E04-68A1-42E8-821D-818E919D4759@gmail.com>
References: 
 <CAKgXx6LWetozLvv6RCHYamD0gXwNvLMi9-NnhM4ghQn=C5+nfA@mail.gmail.com>
 <62C4B740-3DF3-4D48-8664-6BFBE86388C7@gmail.com>
 <CA+ZeGPB7mh5_yLieB+VeA879A=MqP3FH4EgAxhfm-DcDPz07ow@mail.gmail.com>
 <CAJgRgoushtddjUdpqVMT_o0PF2rAKg-oLNGTGz7MXiEiX3v3dA@mail.gmail.com>
 <CAKgXx6Kz7dKg95mCskn9GQnsULR10RUZDh8WPd=755K8WGMgwg@mail.gmail.com>
 <A732B22E-1D47-4512-B876-5FA7A28D1E8F@gmail.com>
 <CAD9ohx-ovgzZ1CnzZKOi+o1TkTZ+MyhddN_ekY-ydOFRyWTAWQ@mail.gmail.com>
 <09772E04-68A1-42E8-821D-818E919D4759@gmail.com>
From: "Cody A. Ray" <cody.a.ray@gmail.com>
Date: Thu, 6 Feb 2014 21:09:13 -0600
Message-ID: 
 <CAOW7FxYxF=7JpFT_GFnnJobOR7+U7KjUwTJkDZO+x_ONuTX8hw@mail.gmail.com>
Subject: Re: http-client version conflict
To: user@storm.incubator.apache.org
Cc: dev@storm.incubator.apache.org
Content-Type: multipart/alternative; boundary=089e0149d0d0ba40a904f1c8519b

--089e0149d0d0ba40a904f1c8519b
Content-Type: text/plain; charset=ISO-8859-1

+1 for relocating storm's dependencies. I just ran into this a couple weeks
ago and have had to side-shelf the project until I find a work-around. Will
check-out the shade plugin. Thanks for the rec!

-Cody


On Thu, Feb 6, 2014 at 9:07 PM, P. Taylor Goetz <ptgoetz@gmail.com> wrote:

> I tend to agree.
>
> The move to maven makes package relocation very easy, but we should be
> careful not to abuse it if that's a direction we decide to take.
>
> IMHO, users having to modify the contents of $STORM_HOME/lib is really bad
> and should only be done as a last resort. If someone asks for help with an
> issue after they've modified the contents of $STORM_HOM/lib, all bets are
> off.
>
> I also agree that we need to keep a close watch on our dependencies, and
> that some might warrant re-evaluation.
>
> I'd love to others opinions.
>
> - Taylor
>
> On Feb 6, 2014, at 9:28 PM, Adam Lewis <mail@adamlewis.com> wrote:
>
> My $0.02 on this subject:
>
>  Without going down the path of class loader or OSGi mania and becoming a
> full container, I'd definitely be in favor of storm relocating its own
> dependencies.  In this way edge cases around things like reflection can be
> handled once within storm rather than burdening every topology builder with
> those details.  Part of the problem seems to be that storm makes extensive
> use (directly or transitively) of a lot of go-to utility libraries like
> guava, thrift, jodatime, json-simple, snakeyaml, commons-io, etc... I'm
> sure that leveraging these libraries allowed storm's development to proceed
> rapidly, but from a maturity perspective, it is problematic to impose these
> version choices on users.  And while I might want Storm to, say, try to
> track the latest Guava version, that same policy could be very problematic
> for others.
>
> If storm can relocate even some of its own dependencies, I think that
> would be a great help to me at least.  Longer term, I wonder how much of
> some of these libraries are really being used.  For example, is clj-time
> (and by extension joda-time) really needed? Or just a small fraction of the
> functionality in that library?  I can probably pitch in some of the effort
> required to do this, if this is the direction people want to go in.
>
>
>
>
> On Thu, Feb 6, 2014 at 8:44 PM, P. Taylor Goetz <ptgoetz@gmail.com> wrote:
>
>> I'm glad the shader plugin worked for you.
>>
>> Updating dependencies can be tricky as it can easily introduce
>> regressions.
>>
>> Ultimately we need to figure out the best solution to avoiding conflicts
>> between user code (i.e. dependencies in topology jar files) and Storm's
>> libraries.
>>
>> The classloader approach has been attempted, but IMO Storm's use of
>> serialization complicates things significantly. Package relocation seems to
>> be a relatively lightweight solution.
>>
>> If that's a direction we pursue, then it introduces the question of
>> whether Storm should relocate its dependencies, or if that should be left
>> up to the user (topology developer).
>>
>> Elastic Search has gone down the path of relocating some of their
>> dependencies [1] (not necessarily an endorsement, just an observation).
>>
>> I've CC'd dev@ since this is all related to the infamous issue #115,
>> which is now STORM-129 [2].
>>
>> - Taylor
>>
>> [1]
>> https://github.com/elasticsearch/elasticsearch/blob/master/pom.xml#L474
>> [2] https://issues.apache.org/jira/browse/STORM-129
>>
>>
>>
>>
>>
>> On Feb 6, 2014, at 7:25 PM, Vinay Pothnis <vinay.pothnis@gmail.com>
>> wrote:
>>
>> Thank you all for replies! The shader-plugin solution seems to work for
>> us.
>>
>> I wonder if we can create a JIRA ticket for storm to upgrade the
>> http-client library as part of their next release.
>>
>> -Vinay
>>
>>
>> On Thu, Feb 6, 2014 at 2:38 PM, Michael Rose <michael@fullcontact.com>wrote:
>>
>>> We've done this with SLF4j and Guava as well without issues.
>>>
>>> Michael Rose (@Xorlev <https://twitter.com/xorlev>)
>>> Senior Platform Engineer, FullContact <http://www.fullcontact.com/>
>>> michael@fullcontact.com
>>>
>>>
>>> On Thu, Feb 6, 2014 at 3:03 PM, Mark Greene <mark@evertrue.com> wrote:
>>>
>>>> We had this problem as well. We modified our chef cookbook to just
>>>> replace the older version with the newer one and storm didn't complain or
>>>> have any other issues as a result.
>>>>
>>>>
>>>> On Wed, Feb 5, 2014 at 10:31 AM, P. Taylor Goetz <ptgoetz@gmail.com>wrote:
>>>>
>>>>> Your best bet is probably  to use the shade plugin to relocate the
>>>>> http-client package so it doesn't conflict with the version storm uses.
>>>>>
>>>>> Storm does this with the libtrhift dependency in storm-core:
>>>>>
>>>>>
>>>>> https://github.com/apache/incubator-storm/blob/master/storm-core/pom.xml#L220
>>>>>
>>>>> (You can ignore the clojure transformer in that config, unless you
>>>>> have non-AOT clojure code that uses the http-client library).
>>>>>
>>>>> More information on using the shade plugin to do package relocations
>>>>> can be found here:
>>>>>
>>>>>
>>>>> http://maven.apache.org/plugins/maven-shade-plugin/examples/class-relocation.html
>>>>>
>>>>> - Taylor
>>>>>
>>>>> On Feb 4, 2014, at 4:27 PM, Vinay Pothnis <vinay.pothnis@gmail.com>
>>>>> wrote:
>>>>>
>>>>> > Hello,
>>>>> >
>>>>> > I am using storm version 0.9.0.1.
>>>>> > My application depends on apache http-client version 4.3.2 - but
>>>>> storm depends on http-client version 4.1.1.
>>>>> >
>>>>> > What is the best way to override this dependency?
>>>>> >
>>>>> > Thanks
>>>>> > Vinay
>>>>>
>>>>>
>>>>
>>>
>>
>>
>
>


-- 
Cody A. Ray, LEED AP
codyaray@drexel.edu
215.501.7891

--089e0149d0d0ba40a904f1c8519b--