hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "York, Zach" <zy...@amazon.com>
Subject Re: [DISCUSS] More Shading
Date Tue, 11 Apr 2017 17:23:30 GMT
Should we allow dependent projects (such as Phoenix) to weigh in on this issue since they are
likely going to be the ones that benefit/are effected?

On 4/11/17, 10:17 AM, "York, Zach" <zyork@amazon.com> wrote:

    +1 (non-binding)
    
    This sounds like a good idea to me!
    
    Zach
    
    On 4/11/17, 9:48 AM, "saint.ack@gmail.com on behalf of Stack" <saint.ack@gmail.com
on behalf of stack@duboce.net> wrote:
    
        Let me revive this thread.
        
        Recall, we are stuck on old or particular versions of critical libs. We are
        unable to update because our versions will clash w/ versions from
        upstreamer hadoop2.7/2.8/3.0/spark, etc. We have a shaded client. We need
        to message downstreamers that they should use it going forward.  This will
        help going forward but it will not inoculate our internals nor an existing
        context where we'd like to be a compatible drop-in.
        
        We could try hackery filtering transitive includes up in poms for each
        version of hadoop/spark that we support but in the end, its a bunch of
        effort, hard to test, and we are unable to dictate the CLASSPATH order in
        all situations.
        
        We could try some shading voodoo inline w/ build. Because shading is a
        post-package step and because we are modularized and shading includes the
        shaded classes in the artifact produced, we'd end up w/ multiple copies of
        guava/netty/etc. classes, an instance per module that makes a reference.
        
        Lets do Sean's idea of a pre-build step where we package and relocate
        ('shade') critical dependencies (Going by the thread above, Ram, Anoop, and
        Andy seems good w/ general idea).
        
        In implementation, we (The HBase PMC) would ask for a new repo [1]. In here
        we'd create a new mvn project. This project would produce a single artifact
        (jar) called hbase-dependencies or hbase-3rdparty or hbase-shaded-3rdparty
        libs. In it would be relocated core libs such as guava and netty (and maybe
        protobuf). We'd publish this artifact and then have hbase depend on it
        changing all references to point at the relocation: e.g. rather than import
        com.google.common.collect.Maps, we'd import
        org.apache.hadoop.hbase.com.google.common.collect.Maps.
        
        We (The HBase PMC) will have to make releases of this new artifact and vote
        on them. I think it will be a relatively rare event.
        
        I'd be up for doing the first cut if folks are game.
        
        St.Ack
        
        
        1. URL via Sean but for committers to view only: https://reporeq.apache.org/
        
        On Sun, Oct 2, 2016 at 10:29 PM, ramkrishna vasudevan <
        ramkrishna.s.vasudevan@gmail.com> wrote:
        
        > +1 for Sean's ideas. Bundling all the dependent libraries and shading them
        > into one jar and HBase referring to it makes sense and should avoid some of
        > the pain in terms of IDE usage. Stack's doc clearly talks about the IDE
        > issues that we may get after this protobuf shading goes in. It may be
        > difficult for new comers and those who don't know this background of why it
        > has to be like that.
        >
        > Regards
        > Ram
        >
        > On Sun, Oct 2, 2016 at 10:51 AM, Stack <stack@duboce.net> wrote:
        >
        > > On Sat, Oct 1, 2016 at 6:32 PM, Jerry He <jerryjch@gmail.com> wrote:
        > >
        > > > How is the proposed going to impact the existing shaded-client and
        > > > shaded-server modules, making them unnecessary and go away?
        > > >
        > >
        > > No. We still need the blanket shading of hbase client and server.
        > >
        > > This effort is about our internals. We have a mess of other components
        > all
        > > up inside us such as HDFS, etc., each with their own sets of dependencies
        > > many of which we have in common. This project t is about making it so we
        > > can upgrade at a rate independent of when our upstreamers choose to
        > change.
        > >
        > >
        > > > It doesn't seem so.  These modules are supposed to shade HBase and
        > > upstream
        > > > from downstream users.
        > > >
        > >
        > > Agree.
        > >
        > > Thanks for drawing out the difference between these two shading efforts,
        > >
        > > St.Ack
        > >
        > >
        > >
        > > > Thanks.
        > > >
        > > > Jerry
        > > >
        > > > On Sat, Oct 1, 2016 at 2:33 PM, Andrew Purtell <
        > andrew.purtell@gmail.com
        > > >
        > > > wrote:
        > > >
        > > > > > Sean has suggested a pre-build step where in another repo
we'd make
        > > > hbase
        > > > > > shaded versions of critical libs, 'release' them (votes,
etc.) and
        > > then
        > > > > > have core depend on these. It be a bunch of work but would
make the
        > > > dev's
        > > > > > life easier.
        > > > >
        > > > > So when we make changes that require updates to and rebuild of
the
        > > > > supporting libraries, as a developer I would make local changes,
        > > install
        > > > a
        > > > > snapshot of that into the local maven cache, then point the HBase
        > build
        > > > at
        > > > > the snapshot, then do the other half of the work, then push up
to
        > both?
        > > > >
        > > > > I think this could work.
        > > >
        > >
        >
        
    
    

Mime
View raw message