hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stack <st...@duboce.net>
Subject Re: Re: [VOTE] hbase-thirdparty project, first release candidate 1.0.0RC0
Date Wed, 05 Jul 2017 03:39:59 GMT
On Tue, Jul 4, 2017 at 6:16 AM, Chunhui Shen <zjusch@163.com> wrote:

> A small thing, we can't get the real version of the thirdparty  library
> after depend on the new generated hbase-shaded-*.jar.
> With the above doubt, I found the file 'META-INF/DEPENDENCIES' which is
> included in hbase-shaded-*.jar,  it contains the information about the
> shaded thirdparty library name and its version.
> However,  'META-INF/DEPENDENCIES' of hbase-shaded-protobuf-1.0.0.jar
> contains none...
>
>
Thanks Chunhui. Makes sense that this should be populated by the
hbase-thirdparty build. Let me fix in next release. I filed HBASE-18321
[hbase-thirdparty] Fix generation of META-INF/DEPENDENCIES to include
dependency list and versions

Thanks,
S


>
> Just realized such a different point,  it's not a problem for most
> developers I think, but better if we could get the real information about
> thirdparty library in hbase project.
>
> At 2017-07-04 13:42:44, "Yu Li" <carp84@gmail.com> wrote:
> >bq. In branch-1, relocate all probobuf and guava references?
> >Yes sir, IMHO this is nice to have (smile). Thanks.
> >
> >Best Regards,
> >Yu
> >
> >On 4 July 2017 at 11:41, Stack <stack@duboce.net> wrote:
> >
> >> On Sun, Jul 2, 2017 at 11:44 PM, Yu Li <carp84@gmail.com> wrote:
> >> >
> >> > bq. No plans to up guava or protobuf versions on branch-1. You think
> we
> >> > should?
> >> > We met with guava conflict when using hbase in Flink, and I could see
> >> > similar issue mentioned in spark. Since branch-1 is still the stable
> >> > release line and will be in the near future, possible to do similar
> >> shading
> >> > w/o upgrading the guava/protobuf version? Thanks.
> >> >
> >> > In branch-1, relocate all probobuf and guava references?
> >>
> >> Thanks Yu Li,
> >> S
> >>
> >>
> >> >
> >> > Best Regards,
> >> > Yu
> >> >
> >> > On 3 July 2017 at 14:01, 张铎(Duo Zhang) <palomino219@gmail.com>
wrote:
> >> >
> >> > > Then it will be a bit pain if developers want to see the source
> code of
> >> > the
> >> > > relocated classes in IDE?
> >> > >
> >> > > 2017-07-03 12:31 GMT+08:00 Stack <stack@duboce.net>:
> >> > >
> >> > > > On Sat, Jul 1, 2017 at 8:16 PM, 张铎(Duo Zhang) <
> palomino219@gmail.com
> >> >
> >> > > > wrote:
> >> > > >
> >> > > > > +1 (binding)
> >> > > > >
> >> > > > > Checked sums and signatures: OK
> >> > > > > 'mvn clean install': OK, the size of the generated jar files
are
> >> > > > reasonable
> >> > > > > Unzip the generated jar files: OK, all classes are relocated
> under
> >> > > > > org/apache/hadoop/hbase/shaded
> >> > > > >
> >> > > > > Only one minor question, seems we do not generate sources
jars?
> >> There
> >> > > is
> >> > > > > a hbase-shaded-protobuf-1.0.0-sources.jar after 'mvn install'
> but
> >> > the
> >> > > > size
> >> > > > > is only 22 bytes.
> >> > > > >
> >> > > > >
> >> > > > Good question. This project is a little bit odd. It is just poms
> >> (and a
> >> > > few
> >> > > > patches). The poms do a download, relocation, and then push to
> deploy
> >> > the
> >> > > > jars to mvn repo. There is no real 'source' -- not anything that
> >> makes
> >> > > > sense to stuff in a src.jar --  other than the poms that are
here
> in
> >> > the
> >> > > > tgz (mvn install puts the jars w/ relocation into your local
> repo....
> >> > > add a
> >> > > > 'deploy' to push to mvn repository -- see README).
> >> > > >
> >> > > > Thanks Duo,
> >> > > > S
> >> > > >
> >> > > >
> >> > > >
> >> > > > > Thanks.
> >> > > > >
> >> > > > > 2017-07-02 2:38 GMT+08:00 Stack <stack@duboce.net>:
> >> > > > >
> >> > > > > > +1 from me.
> >> > > > > > St.Ack
> >> > > > > >
> >> > > > > > On Fri, Jun 30, 2017 at 3:07 PM, Stack <stack@duboce.net>
> wrote:
> >> > > > > >
> >> > > > > > > This is an interesting vote. The vote is on an
RC made out
> of a
> >> > new
> >> > > > > hbase
> >> > > > > > > project, hbase-thirdparty.
> >> > > > > > >
> >> > > > > > > First the usual stuff.
> >> > > > > > >
> >> > > > > > > The 1.0.0RC0 is available at:
> >> > > > > > >
> >> > > > > > >  https://dist.apache.org/repos/dist/dev/hbase/hbase-
> >> > > > > thirdparty/1.0.0RC0/
> >> > > > > > >
> >> > > > > > > Maven artifacts are in the following staging repository:
> >> > > > > > >
> >> > > > > > >  https://repository.apache.org/content/repositories/
> >> > > > > orgapachehbase-1171
> >> > > > > > >
> >> > > > > > > Artifacts are signed with 8ACC93D2 which is at
the tail of
> our
> >> > KEYS
> >> > > > > file
> >> > > > > > > http://www-us.apache.org/dist/hbase/KEYS.
> >> > > > > > >
> >> > > > > > > I tagged this RC as 1.0.0RC at
> 8ffaf3dd561052bcc71772148ecd04
> >> > > > > cdf9e224f3
> >> > > > > > > in the new hbase-thirdparty repository at [1].
> >> > > > > > >
> >> > > > > > > Now to the unusual.
> >> > > > > > >
> >> > > > > > > The artifact was made from a new repository,
> >> hbase-thirdparty[1].
> >> > > > > > > hbase-thirdparty is a project overseen by the
hbase PMC. It
> >> came
> >> > of
> >> > > > > > > discussion on the dev list in the thread '[DISCUSS]
More
> >> Shading'
> >> > > > [2].
> >> > > > > > > Its charge is the hosting of ornery thirdparty
libs -- e.g.
> >> > > protobuf,
> >> > > > > > > guava, netty -- that need patching and/or shading
so we are
> no
> >> > > longer
> >> > > > > > bound
> >> > > > > > > by whatever the version of a lib Hadoop happens
to ship
> (and we
> >> > can
> >> > > > > > change
> >> > > > > > > versions w/o damaging out glorious downstreamers).
In the
> past,
> >> > > when
> >> > > > > > we've
> >> > > > > > > been able, we've done this dirty patching and
shading work
> as
> >> an
> >> > > > > > > unacknowledged 'pre-build' hack step appended
to our
> protobuf
> >> > file
> >> > > > > > > generation. Moving this mess out of our mainline
build to
> live
> >> > > > > elsewhere,
> >> > > > > > > pulled in as a legit 'dependency', simplifies
our build
> >> > > particularly
> >> > > > > > around
> >> > > > > > > the shading of third-part libs; the narrower scope
of a
> >> dedicated
> >> > > > > project
> >> > > > > > > that produces transformed jars is easier to reason
about, is
> >> less
> >> > > > > likely
> >> > > > > > to
> >> > > > > > > clash with existing dependencies whether transitive
or
> >> otherwise,
> >> > > and
> >> > > > > it
> >> > > > > > > makes for less spaghetti poms.
> >> > > > > > >
> >> > > > > > > Other (minor) benefits include our being able
to generate
> >> > protobuf
> >> > > > > files
> >> > > > > > > inline now; no need of the pre-build anymore nor
checking
> in of
> >> > > > > generated
> >> > > > > > > protobuf files (25MB).  There is a patch to remove
them
> >> > (predicated
> >> > > > on
> >> > > > > > this
> >> > > > > > > hbase-thirdparty release): "HBASE-17056 Remove
checked in PB
> >> > > > generated
> >> > > > > > > files".
> >> > > > > > >
> >> > > > > > > Downside is having to explicitly reference our
version in
> code
> >> as
> >> > > in
> >> > > > > > > org.apache.hadoop.hbase.shaded.com.google.protobuf.Message
> >> > rather
> >> > > > than
> >> > > > > > > just com.google.protobuf.Message and so on for
guava
> >> references,
> >> > > etc.
> >> > > > > > > (We've been doing this for a while now so hopefully
it not
> too
> >> > > > strange
> >> > > > > > > developing)*.
> >> > > > > > >
> >> > > > > > > There is little to this new project other than
a few poms
> to do
> >> > > > > packaging
> >> > > > > > > and a couple of patches we apply on build. It
works by
> pulling
> >> > down
> >> > > > > > > dependencies and doing whatever shading, renaming,
or
> patching
> >> we
> >> > > > need.
> >> > > > > > > Currently hbase-thirdparty hosts protobuf (3.1.0
=> 3.3.1),
> >> guava
> >> > > > (12.0
> >> > > > > > =>
> >> > > > > > > 22.0), gson, protobuf-util, and netty 4.1.12-FINAL.
It
> >> publishes
> >> > > > three
> >> > > > > > jars
> >> > > > > > > for hbase to consume: hbase-shaded-protobuf,
> >> hbase-shaded-netty,
> >> > > and
> >> > > > > then
> >> > > > > > > hbase-shaded-miscellaneous for the remaining libs.
> Philosophy
> >> is
> >> > if
> >> > > > the
> >> > > > > > lib
> >> > > > > > > is troublesome, protobuf needs patching and netty
includes
> an
> >> .so
> >> > > > that
> >> > > > > > > needs a rename, then give it its own module (See
README in
> the
> >> > > > release
> >> > > > > > for
> >> > > > > > > more). I've arrived at this layout for hbase-thirdparty
> after a
> >> > > bunch
> >> > > > > of
> >> > > > > > > experimentation and messing with hbase dependencies
(See
> >> > sub-tasks
> >> > > > > under
> >> > > > > > > HBASE-17898 Update dependencies). That said, as
we head to
> >> > > > hbase-2.0.0,
> >> > > > > > we
> >> > > > > > > might have to make a an odd new release of hbase-thirdparty
> but
> >> > > > > > thereafter
> >> > > > > > > I'd imagine it should settle down to be an infrequent
> update.
> >> > > > > > >
> >> > > > > > > VOTE lasts 72 hours. Let +1 be an assent to the
> >> hbase-thirdparty
> >> > > > > project
> >> > > > > > > and a vote for this RC.
> >> > > > > > >
> >> > > > > > > Questions on how it will go w/ this project welcome
on this
> >> > thread.
> >> > > > If
> >> > > > > > > heavy-detail on how the shading works, I suggest
ask on the
> >> > > > '[DISCUSS]
> >> > > > > > More
> >> > > > > > > Shading' thread.
> >> > > > > > >
> >> > > > > > > Thanks,
> >> > > > > > > St.Ack
> >> > > > > > > 1. https://git-wip-us.apache.org/repos/asf/hbase-thirdparty
> >> > > > > > > 2. See the '[DISCUSS] More Shading' thread,
> >> > > > > > http://apache-hbase.679495.n3.
> >> > > > > > > nabble.com/DISCUSS-More-Shading-td4083025.html
> >> > > > > > >
> >> > > > > > > * Doc on how to make use of hbase-thirdparty is
coming for
> the
> >> > > > > refguide.
> >> > > > > > >
> >> > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message