storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vinay Pothnis <vinay.poth...@gmail.com>
Subject Re: http-client version conflict
Date Fri, 07 Feb 2014 03:01:42 GMT
Thanks Taylor and Adam!

I agree with Adam in that it would be better for storm to relocate its
dependencies. I am sure that as storm's usage grows there will be more and
more users with such conflicts. Doing it once and testing well within storm
seems like the best approach.

-Vinay


On Thu, Feb 6, 2014 at 6:28 PM, Adam Lewis <mail@adamlewis.com> wrote:

> My $0.02 on this subject:
>
> Without going down the path of class loader or OSGi mania and becoming a
> full container, I'd definitely be in favor of storm relocating its own
> dependencies.  In this way edge cases around things like reflection can be
> handled once within storm rather than burdening every topology builder with
> those details.  Part of the problem seems to be that storm makes extensive
> use (directly or transitively) of a lot of go-to utility libraries like
> guava, thrift, jodatime, json-simple, snakeyaml, commons-io, etc... I'm
> sure that leveraging these libraries allowed storm's development to proceed
> rapidly, but from a maturity perspective, it is problematic to impose these
> version choices on users.  And while I might want Storm to, say, try to
> track the latest Guava version, that same policy could be very problematic
> for others.
>
> If storm can relocate even some of its own dependencies, I think that
> would be a great help to me at least.  Longer term, I wonder how much of
> some of these libraries are really being used.  For example, is clj-time
> (and by extension joda-time) really needed? Or just a small fraction of the
> functionality in that library?  I can probably pitch in some of the effort
> required to do this, if this is the direction people want to go in.
>
>
>
>
> On Thu, Feb 6, 2014 at 8:44 PM, P. Taylor Goetz <ptgoetz@gmail.com> wrote:
>
>> I'm glad the shader plugin worked for you.
>>
>> Updating dependencies can be tricky as it can easily introduce
>> regressions.
>>
>> Ultimately we need to figure out the best solution to avoiding conflicts
>> between user code (i.e. dependencies in topology jar files) and Storm's
>> libraries.
>>
>> The classloader approach has been attempted, but IMO Storm's use of
>> serialization complicates things significantly. Package relocation seems to
>> be a relatively lightweight solution.
>>
>> If that's a direction we pursue, then it introduces the question of
>> whether Storm should relocate its dependencies, or if that should be left
>> up to the user (topology developer).
>>
>> Elastic Search has gone down the path of relocating some of their
>> dependencies [1] (not necessarily an endorsement, just an observation).
>>
>> I've CC'd dev@ since this is all related to the infamous issue #115,
>> which is now STORM-129 [2].
>>
>> - Taylor
>>
>> [1]
>> https://github.com/elasticsearch/elasticsearch/blob/master/pom.xml#L474
>> [2] https://issues.apache.org/jira/browse/STORM-129
>>
>>
>>
>>
>>
>> On Feb 6, 2014, at 7:25 PM, Vinay Pothnis <vinay.pothnis@gmail.com>
>> wrote:
>>
>> Thank you all for replies! The shader-plugin solution seems to work for
>> us.
>>
>> I wonder if we can create a JIRA ticket for storm to upgrade the
>> http-client library as part of their next release.
>>
>> -Vinay
>>
>>
>> On Thu, Feb 6, 2014 at 2:38 PM, Michael Rose <michael@fullcontact.com>wrote:
>>
>>> We've done this with SLF4j and Guava as well without issues.
>>>
>>> Michael Rose (@Xorlev <https://twitter.com/xorlev>)
>>> Senior Platform Engineer, FullContact <http://www.fullcontact.com/>
>>> michael@fullcontact.com
>>>
>>>
>>> On Thu, Feb 6, 2014 at 3:03 PM, Mark Greene <mark@evertrue.com> wrote:
>>>
>>>> We had this problem as well. We modified our chef cookbook to just
>>>> replace the older version with the newer one and storm didn't complain or
>>>> have any other issues as a result.
>>>>
>>>>
>>>> On Wed, Feb 5, 2014 at 10:31 AM, P. Taylor Goetz <ptgoetz@gmail.com>wrote:
>>>>
>>>>> Your best bet is probably  to use the shade plugin to relocate the
>>>>> http-client package so it doesn't conflict with the version storm uses.
>>>>>
>>>>> Storm does this with the libtrhift dependency in storm-core:
>>>>>
>>>>>
>>>>> https://github.com/apache/incubator-storm/blob/master/storm-core/pom.xml#L220
>>>>>
>>>>> (You can ignore the clojure transformer in that config, unless you
>>>>> have non-AOT clojure code that uses the http-client library).
>>>>>
>>>>> More information on using the shade plugin to do package relocations
>>>>> can be found here:
>>>>>
>>>>>
>>>>> http://maven.apache.org/plugins/maven-shade-plugin/examples/class-relocation.html
>>>>>
>>>>> - Taylor
>>>>>
>>>>> On Feb 4, 2014, at 4:27 PM, Vinay Pothnis <vinay.pothnis@gmail.com>
>>>>> wrote:
>>>>>
>>>>> > Hello,
>>>>> >
>>>>> > I am using storm version 0.9.0.1.
>>>>> > My application depends on apache http-client version 4.3.2 - but
>>>>> storm depends on http-client version 4.1.1.
>>>>> >
>>>>> > What is the best way to override this dependency?
>>>>> >
>>>>> > Thanks
>>>>> > Vinay
>>>>>
>>>>>
>>>>
>>>
>>
>>
>

Mime
View raw message