hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Hsieh <...@cloudera.com>
Subject Re: [DISCUSS] Dependency compatibility
Date Tue, 24 Mar 2015 17:14:09 GMT
On Fri, Mar 20, 2015 at 2:59 PM, Jerry He <jerryjch@gmail.com> wrote:

> On Wed, Mar 18, 2015 at 7:38 AM, Jonathan Hsieh <jon@cloudera.com> wrote:
>
> > We are now considered a mature project and as a "middle turtle" project
> we
> > need to consider the projects that we will inconvenience because
> > they depend on us. All the folks in this discussion are hbase devs and or
> > sophisticated hbase users.  I agree with one of Sean's thoughts -- let us
> > validate if this is a concern for users or not by have a discussion on
> > user@
> > to see how our users feel about it.  Maybe this is no big deal for them.
>
>
> This probably can serve as a head-up for the user community as well.
>
>
> >
> >
> Let's say that this is a concern for the users.  I like Lars's pragmatic
> > critieria -- if the dep change doesn't cause problems for intra version
> > interoperability then, let's let it in and let's update our policy.  If
> it
> > does cause problem's let's keep it out and stick to our guns.
> >
> > To be more concrete there are two scenarios we could test:
> > 1)  an mr job that uses hbase 1.0 client (and it's hadoop dep) needs to
> > work out of the box against 1.1 hbase servers (and it's hadoop deps) or a
> > mixed hbase 1.1- hbase 1.0  environment (e.g. assume we're in the middle
> of
> > rolling upgrading hbase) without failing due to dependency compat checks.
> >
>
> See the coverage matrix below
>
>
> > 2) an mr job that uses hbase 1.0 client (and it's hadoop dep) needs to
> work
> > out of the box against 1.1 hbase servers while the hdfs before and after
> it
> > is upgraded (either rolling or shutdown-restart upgrade)
> > If both succeed despite dep changes, we are in in the clear. If it fails
> > then we do not allow the dep change.
> >
>
> (hbase 1.0 client w/ jackson 1.8 + hadoop 2.4 )  go against (hbase 1.1
> server w/ jackson 1.9 + hadoop 2.4)
>
> (hbase 1.0 client w/ jackson 1.8 + hadoop 2.4 )  go against (hbase 1.1
> server w/ jackson 1.9 + hadoop 2.6)
>
> (hbase 1.0 client w/ jackson 1.8 + hadoop 2.6 )  go against (hbase 1.1
> server w/ jackson 1.9 + hadoop 2.6)
>
> (hbase 1.0 client w/ jackson 1.8 + hadoop 2.4 )  go against (mixed of hbase
> 1.0 w/ jackson 1.8 + hbase 1.1 server w/ Jackson 1.9 + hadoop 2.6)
>
> Is this the coverage you meant?
>
>
> Yes -- with emphasis on the last one because of our rolling upgrade
promise for hbase 1.0->1.1


>
> > To avoid and simplify this problem on our internal builds, we have done
> the
> > exercise of harmonizing all to one version of jackson/guava/etc or
> shading
> > jars for the combination of hadoop and hbase (and other systems) that we
> be
> > ship. I'd like to be in a world where this work is upstream.  At the
> > moment, we don't know whether an upgrade to 2.6 hadoop including its
> > Jackson 1.9 dep will have MR jobs fail or succeed because of classpath
> > ordering issues.  It sounds like it is an may be an issue but we should
> > test to make sure and decide based on that.
> >
> > Jon.
> >
> >
>



-- 
// Jonathan Hsieh (shay)
// HBase Tech Lead, Software Engineer, Cloudera
// jon@cloudera.com // @jmhsieh

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message