hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Hsieh <...@cloudera.com>
Subject Re: [DISCUSS] Dependency compatibility
Date Wed, 18 Mar 2015 14:38:18 GMT
We are now considered a mature project and as a "middle turtle" project we
need to consider the projects that we will inconvenience because
they depend on us. All the folks in this discussion are hbase devs and or
sophisticated hbase users.  I agree with one of Sean's thoughts -- let us
validate if this is a concern for users or not by have a discussion on user@
to see how our users feel about it.  Maybe this is no big deal for them.

Let's say that this is a concern for the users.  I like Lars's pragmatic
critieria -- if the dep change doesn't cause problems for intra version
interoperability then, let's let it in and let's update our policy.  If it
does cause problem's let's keep it out and stick to our guns.

To be more concrete there are two scenarios we could test:
1)  an mr job that uses hbase 1.0 client (and it's hadoop dep) needs to
work out of the box against 1.1 hbase servers (and it's hadoop deps) or a
mixed hbase 1.1- hbase 1.0  environment (e.g. assume we're in the middle of
rolling upgrading hbase) without failing due to dependency compat checks.
2) an mr job that uses hbase 1.0 client (and it's hadoop dep) needs to work
out of the box against 1.1 hbase servers while the hdfs before and after it
is upgraded (either rolling or shutdown-restart upgrade)
If both succeed despite dep changes, we are in in the clear. If it fails
then we do not allow the dep change.

To avoid and simplify this problem on our internal builds, we have done the
exercise of harmonizing all to one version of jackson/guava/etc or shading
jars for the combination of hadoop and hbase (and other systems) that we be
ship. I'd like to be in a world where this work is upstream.  At the
moment, we don't know whether an upgrade to 2.6 hadoop including its
Jackson 1.9 dep will have MR jobs fail or succeed because of classpath
ordering issues.  It sounds like it is an may be an issue but we should
test to make sure and decide based on that.

Jon.

On Sunday, March 15, 2015, lars hofhansl <larsh@apache.org> wrote:

> Sorry for chiming in a bit late.When I wrote up the proposal for our
> dependency story I spent a lot of time thinking about this, we also
> discussed this at length at a PMC meeting. Let's not break our own promise
> in the first ever minor version we release.
>
> Instead of removing that part we can clarify it.
> What we meant to say in that section is that it is pointless to make HBase
> compatibility guarantees if we allow HBase changes that force bringing in a
> dependency that breaks exactly those guarantees.That was mostly around not
> forcing Hadoop to a new *major* version as a dependency for a HBase *minor*
> upgrade. So what that means is that we cannot pull in a new dependency that
> breaks something that we said we would not break.
>
> If we only break "Client Binary Compatibility", which we allow to break in
> a minor upgrade, we'd be OK.
>
> I got lost in the weeds of the various threads here. What will actually
> break when we just upgrade Jackson to 1.9?
> Old clients running against the old HBase jars will continue to work,
> right? Or will MR/Yarn based clients flat due to the different Jackson
> versions between client and server?If old clients work unchanged, we can
> make the change. If not, we can debate whether we have another option, or
> whether MR/Yarn is a different story.
>
Lars
>
>       From: Enis Söztutar <enis.soz@gmail.com>
>  To: "dev@hbase.apache.org" <dev@hbase.apache.org>
>  Sent: Friday, March 13, 2015 11:56 AM
>  Subject: Re: [DISCUSS] Dependency compatibility
>
> >
> >
> > I think we can solve this generally for Hadoop 2.6.0+. There's no reason
> > our HDFS usage should be exposed in the HBase client code, and I think
> the
> > application classpath feature for YARN in that version can isolate us on
> > the MR side. I am willing to do this work in time for 1.1. Realistically
> I
> > don't know the timeline for that version yet. If it turns out the work is
> > more involved or my time is more constrained then I think, I'm willing to
> > accept promise weakening as a practical matter.
> >
>
> HBase-1.x series SHOULD work with Hadoop versions as old as 2.2. That is
> what we promise for 1.x series. So solving the problem in Hadoop-2.6+ will
> not
> solve it for 1.1.
>
> It is great if we can help Hadoop with classloading issues, do shading
> there, and
> do shading in HBase and reduce the dependencies etc. However, since we
> cannot
> do these in HBase-1.x series (we cannot shade deps in the same manner), I
> do
> not see a way how to get around this by doing anything other than what I
> propose.
>
> We have discussed many of the compat dimensions before we adopted them in
> PMC.
> For some of those (like binary compat), we decided that we cannot support
> them
> between minor versions in 1.x , so we decided 'false" on that dimension.
> For these we
> explicitly decided that when we can realistically have more guarantees, we
> can start
> supporting this dimension (client binary compat in minor versions) and have
> 2.x or
> later support those.
>
> I see the dependency compat dimension in the same vein. It is clear (at
> least to me)
> that we cannot support pragmatically any dep compat in 1.x series. If
> you/we can
> make all the necessary changes in Hadoop and HBase, we can reintroduce this
> back. Until then though, I would rather not block any progress and drop the
> support.
> As you said the timeline is not clear, so why are we cornering ourselves
> (especially
> for 1.x series) for this?
>
>
>
>
>
> >
> > I'd be much more comfortable weakening our dependency promises for
> > coprocessor than doing it in general. Folks running coprocessors should
> > already be more risk tolerant and familiar with our internals.
> >
> > For upstreams that don't have the leverage on us of Hadoop, we solve this
> > problem simply by not updating dependencies that we can't trust to not
> > break our downstreams.
> >
> >
> >
> > > I would be disappointed to see a VOTE thread. That means we failed to
> > reach
> > > consensus and needed to fall back to process to resolve differences.
> > >
> > >
> >
> > That's fair. What about the wider audience issue on user@? There's no
> > reason our DISCUSS threads couldn't go there as well.
> >
> >
> >
> > > Why don't we do the doc update and call it a day?
> > >
> >
> > I've been burned by dependency changes in projects I rely on many times
> in
> > the past, usually over changes in code sections that folks didn't think
> > were likely to be used. So I'm very willing to do work now to save
> > downstream users of HBase that same headache.
> >
> > --
> > Sean
> >
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message