Mailing-List: contact dev-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hbase.apache.org
MIME-Version: 1.0
References: <CADcMMgFsODj_bkR5fgfKULf8HinYR_qvbUt2_8RO40anDJpL4Q@mail.gmail.com>
 <35A15B2D-07EF-445A-9CA8-11BF47C72871@gmail.com> <CAHkeaHUhvp7izJQRBNs10P1B+pgXCp4oRXJL3eLEUqk_hbO-Wg@mail.gmail.com>
 <CADcMMgFX06Y_VNQQN=qtyuANORQfjN51OTLnw0-we1yTVve3OQ@mail.gmail.com>
 <CAAT7MkpiuPqYHY8ziFxqAuCzr3X_DgDERPVOowcw3kYLCGivJw@mail.gmail.com>
 <CADcMMgF5P+M7xfC-QU2Em4T-U2aF-+AJnvd6Wd5J4pzJZBNFHw@mail.gmail.com>
 <7AE04EB4-7599-4181-B71D-B261773BF636@amazon.com> <A4F44986-E392-488E-A21F-BF45D01B440C@amazon.com>
 <CAB5sDNLw47rT_EnjkbyxBeFMGZJqLwZ5q28R65dFjtegiAD1LA@mail.gmail.com> <CAN5cbe7E2fBxY_OU3nZvP3rWEPwT9cNGab0T0AtaeyU0LzHHsA@mail.gmail.com>
In-Reply-To: <CAN5cbe7E2fBxY_OU3nZvP3rWEPwT9cNGab0T0AtaeyU0LzHHsA@mail.gmail.com>
From: Jesse Yates <jesse.k.yates@gmail.com>
Date: Tue, 11 Apr 2017 19:34:05 +0000
Message-ID: <CAB5sDNLpwQqoxm5rQvOoO_Ue18KupqG7f3zwrMFHJDWRLcTa1Q@mail.gmail.com>
Subject: Re: [DISCUSS] More Shading
To: dev@hbase.apache.org
Content-Type: multipart/alternative; boundary=001a113f8e306a7aef054ce92d4f
archived-at: Tue, 11 Apr 2017 19:34:24 -0000

--001a113f8e306a7aef054ce92d4f
Content-Type: text/plain; charset=UTF-8

Right, hence the install/package phase (probably with -DskipTests) first.
Probably only have to do this occasionally, as dependencies change.

Agree the IDEs are really unhappy with this though.

Seems like more headache to create another repo, but I'm not too tied
either way. Just asking. Thanks Sean.

-J

On Tue, Apr 11, 2017 at 12:24 PM Sean Busbey <busbey@apache.org> wrote:

> A new module probably won't work due to the fact that we need to reference
> the relocated classes in source code and maven won't do that until the
> "package" phase.
>
> IDEs in particular will barf all over the place.
>
> On Tue, Apr 11, 2017 at 1:04 PM Jesse Yates <jesse.k.yates@gmail.com>
> wrote:
>
> > > would ask for a new repo [1]. In here we'd create a new mvn project.
> >
> > Why get a new repo? A different (new) HBase mvn module that is depended
> > upon via other modules should cover it, IIRC. That module can handle all
> > the shading and not include transitive dependencies. Then in "downstream
> > modules" you should be able to just use the shaded classes. Building
> would
> > require doing a 'mvn install', but that's nothing new.
> >
> > If this was going to support the client I'd be concerned with size of the
> > resulting jar, with all the potential dependencies, but meh - its the
> > server only!
> >
> > Just my $0.02,
> > Jesse
> >
> > On Tue, Apr 11, 2017 at 10:23 AM York, Zach <zyork@amazon.com> wrote:
> >
> > > Should we allow dependent projects (such as Phoenix) to weigh in on
> this
> > > issue since they are likely going to be the ones that benefit/are
> > effected?
> > >
> > > On 4/11/17, 10:17 AM, "York, Zach" <zyork@amazon.com> wrote:
> > >
> > >     +1 (non-binding)
> > >
> > >     This sounds like a good idea to me!
> > >
> > >     Zach
> > >
> > >     On 4/11/17, 9:48 AM, "saint.ack@gmail.com on behalf of Stack" <
> > > saint.ack@gmail.com on behalf of stack@duboce.net> wrote:
> > >
> > >         Let me revive this thread.
> > >
> > >         Recall, we are stuck on old or particular versions of critical
> > > libs. We are
> > >         unable to update because our versions will clash w/ versions
> from
> > >         upstreamer hadoop2.7/2.8/3.0/spark, etc. We have a shaded
> client.
> > > We need
> > >         to message downstreamers that they should use it going forward.
> > > This will
> > >         help going forward but it will not inoculate our internals nor
> an
> > > existing
> > >         context where we'd like to be a compatible drop-in.
> > >
> > >         We could try hackery filtering transitive includes up in poms
> for
> > > each
> > >         version of hadoop/spark that we support but in the end, its a
> > > bunch of
> > >         effort, hard to test, and we are unable to dictate the
> CLASSPATH
> > > order in
> > >         all situations.
> > >
> > >         We could try some shading voodoo inline w/ build. Because
> shading
> > > is a
> > >         post-package step and because we are modularized and shading
> > > includes the
> > >         shaded classes in the artifact produced, we'd end up w/
> multiple
> > > copies of
> > >         guava/netty/etc. classes, an instance per module that makes a
> > > reference.
> > >
> > >         Lets do Sean's idea of a pre-build step where we package and
> > > relocate
> > >         ('shade') critical dependencies (Going by the thread above,
> Ram,
> > > Anoop, and
> > >         Andy seems good w/ general idea).
> > >
> > >         In implementation, we (The HBase PMC) would ask for a new repo
> > > [1]. In here
> > >         we'd create a new mvn project. This project would produce a
> > single
> > > artifact
> > >         (jar) called hbase-dependencies or hbase-3rdparty or
> > > hbase-shaded-3rdparty
> > >         libs. In it would be relocated core libs such as guava and
> netty
> > > (and maybe
> > >         protobuf). We'd publish this artifact and then have hbase
> depend
> > > on it
> > >         changing all references to point at the relocation: e.g. rather
> > > than import
> > >         com.google.common.collect.Maps, we'd import
> > >         org.apache.hadoop.hbase.com.google.common.collect.Maps.
> > >
> > >         We (The HBase PMC) will have to make releases of this new
> > artifact
> > > and vote
> > >         on them. I think it will be a relatively rare event.
> > >
> > >         I'd be up for doing the first cut if folks are game.
> > >
> > >         St.Ack
> > >
> > >
> > >         1. URL via Sean but for committers to view only:
> > > https://reporeq.apache.org/
> > >
> > >         On Sun, Oct 2, 2016 at 10:29 PM, ramkrishna vasudevan <
> > >         ramkrishna.s.vasudevan@gmail.com> wrote:
> > >
> > >         > +1 for Sean's ideas. Bundling all the dependent libraries and
> > > shading them
> > >         > into one jar and HBase referring to it makes sense and should
> > > avoid some of
> > >         > the pain in terms of IDE usage. Stack's doc clearly talks
> about
> > > the IDE
> > >         > issues that we may get after this protobuf shading goes in.
> It
> > > may be
> > >         > difficult for new comers and those who don't know this
> > > background of why it
> > >         > has to be like that.
> > >         >
> > >         > Regards
> > >         > Ram
> > >         >
> > >         > On Sun, Oct 2, 2016 at 10:51 AM, Stack <stack@duboce.net>
> > wrote:
> > >         >
> > >         > > On Sat, Oct 1, 2016 at 6:32 PM, Jerry He <
> jerryjch@gmail.com
> > >
> > > wrote:
> > >         > >
> > >         > > > How is the proposed going to impact the existing
> > > shaded-client and
> > >         > > > shaded-server modules, making them unnecessary and go
> away?
> > >         > > >
> > >         > >
> > >         > > No. We still need the blanket shading of hbase client and
> > > server.
> > >         > >
> > >         > > This effort is about our internals. We have a mess of other
> > > components
> > >         > all
> > >         > > up inside us such as HDFS, etc., each with their own sets
> of
> > > dependencies
> > >         > > many of which we have in common. This project t is about
> > > making it so we
> > >         > > can upgrade at a rate independent of when our upstreamers
> > > choose to
> > >         > change.
> > >         > >
> > >         > >
> > >         > > > It doesn't seem so.  These modules are supposed to shade
> > > HBase and
> > >         > > upstream
> > >         > > > from downstream users.
> > >         > > >
> > >         > >
> > >         > > Agree.
> > >         > >
> > >         > > Thanks for drawing out the difference between these two
> > > shading efforts,
> > >         > >
> > >         > > St.Ack
> > >         > >
> > >         > >
> > >         > >
> > >         > > > Thanks.
> > >         > > >
> > >         > > > Jerry
> > >         > > >
> > >         > > > On Sat, Oct 1, 2016 at 2:33 PM, Andrew Purtell <
> > >         > andrew.purtell@gmail.com
> > >         > > >
> > >         > > > wrote:
> > >         > > >
> > >         > > > > > Sean has suggested a pre-build step where in another
> > > repo we'd make
> > >         > > > hbase
> > >         > > > > > shaded versions of critical libs, 'release' them
> > (votes,
> > > etc.) and
> > >         > > then
> > >         > > > > > have core depend on these. It be a bunch of work but
> > > would make the
> > >         > > > dev's
> > >         > > > > > life easier.
> > >         > > > >
> > >         > > > > So when we make changes that require updates to and
> > > rebuild of the
> > >         > > > > supporting libraries, as a developer I would make local
> > > changes,
> > >         > > install
> > >         > > > a
> > >         > > > > snapshot of that into the local maven cache, then point
> > > the HBase
> > >         > build
> > >         > > > at
> > >         > > > > the snapshot, then do the other half of the work, then
> > > push up to
> > >         > both?
> > >         > > > >
> > >         > > > > I think this could work.
> > >         > > >
> > >         > >
> > >         >
> > >
> > >
> > >
> > >
> > > --
> > Jesse Yates
> > Founder/CEO Fineo.io
> > Book a meeting: https://calendly.com/jyates
> >
>
-- 
Jesse Yates
Founder/CEO Fineo.io
Book a meeting: https://calendly.com/jyates

--001a113f8e306a7aef054ce92d4f--