hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean Busbey <bus...@cloudera.com>
Subject Re: [DISCUSS] Dependency compatibility
Date Wed, 11 Mar 2015 23:13:31 GMT
ugh, yes please let's not return to the days of hadoop-version-specific
HBase builds.

I'm just spinning up on isolating third-party dependencies over in Hadoop.
I'd be happy to apply whatever works there within HBase. This only really
helps us for HBase 2.y though, right?

Without sidetracking this thread too much, before I file a jira for
isolating third-party deps what scope are we looking for? hbase-client and
the MR tooling only? i.e I'd prefer to state that folks making use of
LimitedPrivate features like coprocessors don't get isolation from our
dependencies.


On Wed, Mar 11, 2015 at 5:44 PM, Nick Dimiduk <ndimiduk@gmail.com> wrote:

> The advantage of shading/JarJaring our dependencies is we *don't* get stuck
> with hbase-A.B-hadoop-Z.Y. Those releases are a hassle for users, and a
> hassle for release managers. The disadvantage being runtime bloat in the
> form of an excessive number of classes to load.
>
> I think there's a middle ground where we shade only those with a bad
> track-record. Enis has a good handle on this list.
>
> On Wed, Mar 11, 2015 at 3:38 PM, Enis Söztutar <enis.soz@gmail.com> wrote:
>
> > >
> > >
> > > Can we do something even more generic like shade [1] our dependencies
> for
> > > some of the common sources of dependency pain like guava, jackson, and
> > > netty?  This way we decouple hbase's pain from hadoop's and allow
> clients
> > > to choose their own versions of libs to use.  We'd likely still cause
> > some
> > > inconvenience for 1.0 ->1.1 but this should eliminate this problem from
> > > biting us again if we are on hadopo 3.x->3.x+1 and hbase 2.y and hbase
> > > 2.y+1.
> > >
> >
> > I think shading some of the dependencies is a good idea. Obvious stuff
> are
> > protobuf, netty, guava, jackson, etc.
> > We should also convince Hadoop to do it as well. We should decouple our
> > guava and
> > protobuf version from that of Hadoop's. I've read that elastic search
> uses
> > maven shade
> > plugin, we can look at what they have done.
> >
> >
> > >
> > > Jon.
> > >
> > >
> > > [1] http://maven.apache.org/plugins/maven-shade-plugin/
> > >
> > > On Wed, Mar 11, 2015 at 2:04 PM, Enis Söztutar <enis@apache.org>
> wrote:
> > >
> > > > Over at https://issues.apache.org/jira/browse/HBASE-13149, there is
> > some
> > > > discussion about changing the jackson version from 1.8 to 1.9 that is
> > > worth
> > > > bringing here because of the wider audience. The discussion is about
> > > > jackson version specific in this issue, but we should also consider
> > > future
> > > > changes to libs.
> > > >
> > > > The question is, whether we can change the dep version of jackson
> from
> > > 1.8
> > > > to 1.9 in HBase-1.1. According to our guidelines, we said that we
> will
> > > not
> > > > do breaking changes to our deps in minor versions. From the analysis
> of
> > > > jackson compat attached at jira, it seems that jackson have some
> > changes
> > > > breaking BC.
> > > >
> > > > So, from a purist perspective, if we follow our guidelines literally,
> > we
> > > > should not make this change. However, I think we should make that
> > change
> > > in
> > > > 1.1 (and make similar changes in the future as well), because:
> > > >
> > > > 1. Hadoop upgraded its version of jackson in 2.5, and HBase MR jobs
> > fail
> > > > flat out without corresponding change in HBase (with Hadoop-2.5+). We
> > do
> > > > not have any control over Hadoop or similar core dependencies. We
> > cannot
> > > be
> > > > realistically expected to guarantee more than what they do. I would
> > > rather
> > > > do the change in 1.1 and not inconvenience the users, than not do the
> > > > change, and have to explain how to replace the jars in the docs. This
> > > > argument should be extended beyond the jackson dependency.
> > > >
> > > > 2. HBase is not at the maturity level for following the dependency
> > compat
> > > > guide literally without reducing the dep footprint, better isolation
> of
> > > MR
> > > > / Mini cluster out of hbase-server and possible participation from
> > > hadoop.
> > > > We cannot easily bump up the major version for these as well, since
> > major
> > > > version implies other things.
> > > >
> > > > So, my proposal is:
> > > >  - Commit HBASE-13149 to master and 1.1
> > > >  - Either change the dependency compat story for minor versions to
> > false,
> > > > or add a footnote saying that there may be exceptions because of the
> > > > reasons listed above.
> > > >
> > > > Let me know what you guys think.
> > > > Enis
> > > >
> > >
> > >
> > >
> > > --
> > > // Jonathan Hsieh (shay)
> > > // HBase Tech Lead, Software Engineer, Cloudera
> > > // jon@cloudera.com // @jmhsieh
> > >
> >
>



-- 
Sean

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message