hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Enis Söztutar <enis....@gmail.com>
Subject Re: [DISCUSS] Dependency compatibility
Date Thu, 12 Mar 2015 18:20:41 GMT
This is good discussion, but I would like to reach a consensus and move on
with HBASE-13149.

My conclusion from the thread is that we cannot be realistically expected
to keep dependency compat between minor versions because of lack of shading
in HBase and Hadoop, and our dependencies are themselves not semver, and we
cannot promise more than our dependencies promise.

So I would like to formally drop support for dependency compat between
minor versions as defined in
https://hbase.apache.org/book.html#hbase.versioning. We can reintroduce
later when we have better isolation/guarantees. In the mean time, we can
move on.

The PMC has approved the compat guide, but I am not sure whether we need a
VOTE thraed. What do you guys think?

Enis

On Wed, Mar 11, 2015 at 11:32 PM, Andrew Purtell <apurtell@apache.org>
wrote:

> When did we retire 0.98 (grin)
>
> Seriously, let's not have Hadoop version specific builds beyond 0.98. Every
> other week or so we find and fix breakage in the Hadoop 1 build because
> nobody develops with Hadoop 1 or runs it in production. The combinatorics
> will be problematic starting with only two branches based on this
> experience.
>
> I don't see how coprocessors can be isolated from our dependencies unless
> we implement classloader/environment support for that, which would be a lot
> of work and yet not isolate them from other changes they'll face due to
> proximity to other low level details.
>
>
> On Wednesday, March 11, 2015, Sean Busbey <busbey@cloudera.com> wrote:
>
> > ugh, yes please let's not return to the days of hadoop-version-specific
> > HBase builds.
> >
> > I'm just spinning up on isolating third-party dependencies over in
> Hadoop.
> > I'd be happy to apply whatever works there within HBase. This only really
> > helps us for HBase 2.y though, right?
> >
> > Without sidetracking this thread too much, before I file a jira for
> > isolating third-party deps what scope are we looking for? hbase-client
> and
> > the MR tooling only? i.e I'd prefer to state that folks making use of
> > LimitedPrivate features like coprocessors don't get isolation from our
> > dependencies.
> >
> >
> > On Wed, Mar 11, 2015 at 5:44 PM, Nick Dimiduk <ndimiduk@gmail.com
> > <javascript:;>> wrote:
> >
> > > The advantage of shading/JarJaring our dependencies is we *don't* get
> > stuck
> > > with hbase-A.B-hadoop-Z.Y. Those releases are a hassle for users, and a
> > > hassle for release managers. The disadvantage being runtime bloat in
> the
> > > form of an excessive number of classes to load.
> > >
> > > I think there's a middle ground where we shade only those with a bad
> > > track-record. Enis has a good handle on this list.
> > >
> > > On Wed, Mar 11, 2015 at 3:38 PM, Enis Söztutar <enis.soz@gmail.com
> > <javascript:;>> wrote:
> > >
> > > > >
> > > > >
> > > > > Can we do something even more generic like shade [1] our
> dependencies
> > > for
> > > > > some of the common sources of dependency pain like guava, jackson,
> > and
> > > > > netty?  This way we decouple hbase's pain from hadoop's and allow
> > > clients
> > > > > to choose their own versions of libs to use.  We'd likely still
> cause
> > > > some
> > > > > inconvenience for 1.0 ->1.1 but this should eliminate this problem
> > from
> > > > > biting us again if we are on hadopo 3.x->3.x+1 and hbase 2.y and
> > hbase
> > > > > 2.y+1.
> > > > >
> > > >
> > > > I think shading some of the dependencies is a good idea. Obvious
> stuff
> > > are
> > > > protobuf, netty, guava, jackson, etc.
> > > > We should also convince Hadoop to do it as well. We should decouple
> our
> > > > guava and
> > > > protobuf version from that of Hadoop's. I've read that elastic search
> > > uses
> > > > maven shade
> > > > plugin, we can look at what they have done.
> > > >
> > > >
> > > > >
> > > > > Jon.
> > > > >
> > > > >
> > > > > [1] http://maven.apache.org/plugins/maven-shade-plugin/
> > > > >
> > > > > On Wed, Mar 11, 2015 at 2:04 PM, Enis Söztutar <enis@apache.org
> > <javascript:;>>
> > > wrote:
> > > > >
> > > > > > Over at https://issues.apache.org/jira/browse/HBASE-13149, there
> > is
> > > > some
> > > > > > discussion about changing the jackson version from 1.8 to 1.9
> that
> > is
> > > > > worth
> > > > > > bringing here because of the wider audience. The discussion
is
> > about
> > > > > > jackson version specific in this issue, but we should also
> consider
> > > > > future
> > > > > > changes to libs.
> > > > > >
> > > > > > The question is, whether we can change the dep version of jackson
> > > from
> > > > > 1.8
> > > > > > to 1.9 in HBase-1.1. According to our guidelines, we said that
we
> > > will
> > > > > not
> > > > > > do breaking changes to our deps in minor versions. From the
> > analysis
> > > of
> > > > > > jackson compat attached at jira, it seems that jackson have
some
> > > > changes
> > > > > > breaking BC.
> > > > > >
> > > > > > So, from a purist perspective, if we follow our guidelines
> > literally,
> > > > we
> > > > > > should not make this change. However, I think we should make
that
> > > > change
> > > > > in
> > > > > > 1.1 (and make similar changes in the future as well), because:
> > > > > >
> > > > > > 1. Hadoop upgraded its version of jackson in 2.5, and HBase
MR
> jobs
> > > > fail
> > > > > > flat out without corresponding change in HBase (with
> Hadoop-2.5+).
> > We
> > > > do
> > > > > > not have any control over Hadoop or similar core dependencies.
We
> > > > cannot
> > > > > be
> > > > > > realistically expected to guarantee more than what they do.
I
> would
> > > > > rather
> > > > > > do the change in 1.1 and not inconvenience the users, than not
do
> > the
> > > > > > change, and have to explain how to replace the jars in the docs.
> > This
> > > > > > argument should be extended beyond the jackson dependency.
> > > > > >
> > > > > > 2. HBase is not at the maturity level for following the
> dependency
> > > > compat
> > > > > > guide literally without reducing the dep footprint, better
> > isolation
> > > of
> > > > > MR
> > > > > > / Mini cluster out of hbase-server and possible participation
> from
> > > > > hadoop.
> > > > > > We cannot easily bump up the major version for these as well,
> since
> > > > major
> > > > > > version implies other things.
> > > > > >
> > > > > > So, my proposal is:
> > > > > >  - Commit HBASE-13149 to master and 1.1
> > > > > >  - Either change the dependency compat story for minor versions
> to
> > > > false,
> > > > > > or add a footnote saying that there may be exceptions because
of
> > the
> > > > > > reasons listed above.
> > > > > >
> > > > > > Let me know what you guys think.
> > > > > > Enis
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > // Jonathan Hsieh (shay)
> > > > > // HBase Tech Lead, Software Engineer, Cloudera
> > > > > // jon@cloudera.com <javascript:;> // @jmhsieh
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > Sean
> >
>
>
> --
> Best regards,
>
>    - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message