hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nick Dimiduk <ndimi...@gmail.com>
Subject Re: [DISCUSS] Dependency compatibility
Date Thu, 12 Mar 2015 18:38:39 GMT
Always the pragmatist. I accept your proposal with acknowledgement of an
effort to improve the state of things going forward.

On Thursday, March 12, 2015, Enis Söztutar <enis.soz@gmail.com> wrote:

> This is good discussion, but I would like to reach a consensus and move on
> with HBASE-13149.
>
> My conclusion from the thread is that we cannot be realistically expected
> to keep dependency compat between minor versions because of lack of shading
> in HBase and Hadoop, and our dependencies are themselves not semver, and we
> cannot promise more than our dependencies promise.
>
> So I would like to formally drop support for dependency compat between
> minor versions as defined in
> https://hbase.apache.org/book.html#hbase.versioning. We can reintroduce
> later when we have better isolation/guarantees. In the mean time, we can
> move on.
>
> The PMC has approved the compat guide, but I am not sure whether we need a
> VOTE thraed. What do you guys think?
>
> Enis
>
> On Wed, Mar 11, 2015 at 11:32 PM, Andrew Purtell <apurtell@apache.org
> <javascript:;>>
> wrote:
>
> > When did we retire 0.98 (grin)
> >
> > Seriously, let's not have Hadoop version specific builds beyond 0.98.
> Every
> > other week or so we find and fix breakage in the Hadoop 1 build because
> > nobody develops with Hadoop 1 or runs it in production. The combinatorics
> > will be problematic starting with only two branches based on this
> > experience.
> >
> > I don't see how coprocessors can be isolated from our dependencies unless
> > we implement classloader/environment support for that, which would be a
> lot
> > of work and yet not isolate them from other changes they'll face due to
> > proximity to other low level details.
> >
> >
> > On Wednesday, March 11, 2015, Sean Busbey <busbey@cloudera.com
> <javascript:;>> wrote:
> >
> > > ugh, yes please let's not return to the days of hadoop-version-specific
> > > HBase builds.
> > >
> > > I'm just spinning up on isolating third-party dependencies over in
> > Hadoop.
> > > I'd be happy to apply whatever works there within HBase. This only
> really
> > > helps us for HBase 2.y though, right?
> > >
> > > Without sidetracking this thread too much, before I file a jira for
> > > isolating third-party deps what scope are we looking for? hbase-client
> > and
> > > the MR tooling only? i.e I'd prefer to state that folks making use of
> > > LimitedPrivate features like coprocessors don't get isolation from our
> > > dependencies.
> > >
> > >
> > > On Wed, Mar 11, 2015 at 5:44 PM, Nick Dimiduk <ndimiduk@gmail.com
> <javascript:;>
> > > <javascript:;>> wrote:
> > >
> > > > The advantage of shading/JarJaring our dependencies is we *don't* get
> > > stuck
> > > > with hbase-A.B-hadoop-Z.Y. Those releases are a hassle for users,
> and a
> > > > hassle for release managers. The disadvantage being runtime bloat in
> > the
> > > > form of an excessive number of classes to load.
> > > >
> > > > I think there's a middle ground where we shade only those with a bad
> > > > track-record. Enis has a good handle on this list.
> > > >
> > > > On Wed, Mar 11, 2015 at 3:38 PM, Enis Söztutar <enis.soz@gmail.com
> <javascript:;>
> > > <javascript:;>> wrote:
> > > >
> > > > > >
> > > > > >
> > > > > > Can we do something even more generic like shade [1] our
> > dependencies
> > > > for
> > > > > > some of the common sources of dependency pain like guava,
> jackson,
> > > and
> > > > > > netty?  This way we decouple hbase's pain from hadoop's and
allow
> > > > clients
> > > > > > to choose their own versions of libs to use.  We'd likely still
> > cause
> > > > > some
> > > > > > inconvenience for 1.0 ->1.1 but this should eliminate this
> problem
> > > from
> > > > > > biting us again if we are on hadopo 3.x->3.x+1 and hbase
2.y and
> > > hbase
> > > > > > 2.y+1.
> > > > > >
> > > > >
> > > > > I think shading some of the dependencies is a good idea. Obvious
> > stuff
> > > > are
> > > > > protobuf, netty, guava, jackson, etc.
> > > > > We should also convince Hadoop to do it as well. We should decouple
> > our
> > > > > guava and
> > > > > protobuf version from that of Hadoop's. I've read that elastic
> search
> > > > uses
> > > > > maven shade
> > > > > plugin, we can look at what they have done.
> > > > >
> > > > >
> > > > > >
> > > > > > Jon.
> > > > > >
> > > > > >
> > > > > > [1] http://maven.apache.org/plugins/maven-shade-plugin/
> > > > > >
> > > > > > On Wed, Mar 11, 2015 at 2:04 PM, Enis Söztutar <enis@apache.org
> <javascript:;>
> > > <javascript:;>>
> > > > wrote:
> > > > > >
> > > > > > > Over at https://issues.apache.org/jira/browse/HBASE-13149,
> there
> > > is
> > > > > some
> > > > > > > discussion about changing the jackson version from 1.8
to 1.9
> > that
> > > is
> > > > > > worth
> > > > > > > bringing here because of the wider audience. The discussion
is
> > > about
> > > > > > > jackson version specific in this issue, but we should also
> > consider
> > > > > > future
> > > > > > > changes to libs.
> > > > > > >
> > > > > > > The question is, whether we can change the dep version
of
> jackson
> > > > from
> > > > > > 1.8
> > > > > > > to 1.9 in HBase-1.1. According to our guidelines, we said
that
> we
> > > > will
> > > > > > not
> > > > > > > do breaking changes to our deps in minor versions. From
the
> > > analysis
> > > > of
> > > > > > > jackson compat attached at jira, it seems that jackson
have
> some
> > > > > changes
> > > > > > > breaking BC.
> > > > > > >
> > > > > > > So, from a purist perspective, if we follow our guidelines
> > > literally,
> > > > > we
> > > > > > > should not make this change. However, I think we should
make
> that
> > > > > change
> > > > > > in
> > > > > > > 1.1 (and make similar changes in the future as well), because:
> > > > > > >
> > > > > > > 1. Hadoop upgraded its version of jackson in 2.5, and HBase
MR
> > jobs
> > > > > fail
> > > > > > > flat out without corresponding change in HBase (with
> > Hadoop-2.5+).
> > > We
> > > > > do
> > > > > > > not have any control over Hadoop or similar core dependencies.
> We
> > > > > cannot
> > > > > > be
> > > > > > > realistically expected to guarantee more than what they
do. I
> > would
> > > > > > rather
> > > > > > > do the change in 1.1 and not inconvenience the users, than
not
> do
> > > the
> > > > > > > change, and have to explain how to replace the jars in
the
> docs.
> > > This
> > > > > > > argument should be extended beyond the jackson dependency.
> > > > > > >
> > > > > > > 2. HBase is not at the maturity level for following the
> > dependency
> > > > > compat
> > > > > > > guide literally without reducing the dep footprint, better
> > > isolation
> > > > of
> > > > > > MR
> > > > > > > / Mini cluster out of hbase-server and possible participation
> > from
> > > > > > hadoop.
> > > > > > > We cannot easily bump up the major version for these as
well,
> > since
> > > > > major
> > > > > > > version implies other things.
> > > > > > >
> > > > > > > So, my proposal is:
> > > > > > >  - Commit HBASE-13149 to master and 1.1
> > > > > > >  - Either change the dependency compat story for minor
versions
> > to
> > > > > false,
> > > > > > > or add a footnote saying that there may be exceptions because
> of
> > > the
> > > > > > > reasons listed above.
> > > > > > >
> > > > > > > Let me know what you guys think.
> > > > > > > Enis
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > // Jonathan Hsieh (shay)
> > > > > > // HBase Tech Lead, Software Engineer, Cloudera
> > > > > > // jon@cloudera.com <javascript:;> <javascript:;>
// @jmhsieh
> > > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Sean
> > >
> >
> >
> > --
> > Best regards,
> >
> >    - Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
> > (via Tom White)
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message