Mailing-List: contact yarn-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: yarn-dev@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of eli@cloudera.com designates
 209.85.216.169 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <75143F26-F375-4FF7-9B62-A5AF85974593@hortonworks.com>
References: <F4A9CA29-6536-4F3D-95B4-9BF18F14C8DE@hortonworks.com>
	<CAPn_vTs=D9tkUW6CknJHtiBC1GXw1dQrmiTq0EtXu530VgJ1dQ@mail.gmail.com>
	<75143F26-F375-4FF7-9B62-A5AF85974593@hortonworks.com>
Date: Thu, 31 Jan 2013 11:51:13 -0800
Message-ID: 
 <CAPn_vTvXc7tLo-ckzDA28eD4v7Xi+sZN6ECP-Vsnk7RNV5Cjog@mail.gmail.com>
Subject: Re: Release numbering for branch-2 releases
From: Eli Collins <eli@cloudera.com>
To: "hdfs-dev@hadoop.apache.org" <hdfs-dev@hadoop.apache.org>
Cc: "mapreduce-dev@hadoop.apache.org" <mapreduce-dev@hadoop.apache.org>,
 yarn-dev@hadoop.apache.org,
	"common-dev@hadoop.apache.org" <common-dev@hadoop.apache.org>
Content-Type: multipart/alternative; boundary=20cf303b4219015f0c04d49af35a

--20cf303b4219015f0c04d49af35a
Content-Type: text/plain; charset=ISO-8859-1

We also need to spell out what's permissible *before* GA as well.  The
alpha/beta labels, as I understand them, are not green lights to break
anything as long as it's not API compatibility.  The API compatibility
story has been somewhat fuzzy as well, eg MR2 requires users recompile all
their Hadoop 1.x jobs (ouch).  We've been working on stabilizing 2.x for a
while now and we need to start slating some changes to 3.x if we want to
get a 2.x GA release out soon.  To do that we have to consider issues for
end users (and downstream projects) upgrading from 0.23 releases and older
2.0.x releases, aside from just API compatibility, in terms of what's
permissible in the releases between now and GA.

Thanks,
Eli

On Wed, Jan 30, 2013 at 5:10 PM, Arun C Murthy <acm@hortonworks.com> wrote:

> The discussions in HADOOP-9151 were related to wire-compatibility. I think
> we all agree that breaking API compatibility is not allowed without
> deprecating them first in a prior major release - this is something we have
> followed since hadoop-0.1.
>
> I agree we need to spell out what changes we can and cannot do *after* we
> go GA, for e.g.:
> # Clearly incompatible *API* changes are *not* allowed in hadoop-2 post-GA.
> # Do we allow incompatible changes on Client-Server protocols? I would say
> *no*.
> # Do we allow incompatible changes on internal-server protocols (for e.g.
> NN-DN or NN-NN in HA setup or RM-NM in YARN) to ensure we support
> rolling-upgrades? I would like to not allow this, but I do not know how
> feasible this is. An option is to allow these changes between minor
> releases i.e. between hadoop-2.10 and hadoop-2.11.
> # Do we allow changes which force a HDFS metadata upgrade between a minor
> upgrade i.e. hadoop-2.20 to hadoop-2.21?
> # Clearly *no* incompatible changes (API/client-server/server-server)
> changes are allowed in a patch release i.e. hadoop-2.20.0 and hadoop-2.20.1
> have to be compatible among all respects.
>
> What else am I missing?
>
> I'll make sure we update our Roadmap wiki and other docs post this
> discussion.
>
> thanks,
> Arun
>
>
>
> On Jan 30, 2013, at 4:21 PM, Eli Collins wrote:
>
> > Thanks for bringing this up Arun.  One of the issues is that we
> > haven't been clear about what type of compatibility breakages are
> > allowed, and which are not.  For example, renaming FileSystem#open is
> > incompatible, and not OK, regardless of the alpha/beta tag.  Breaking
> > a server/server APIs is OK pre-GA but probably not post GA, at least
> > in a point release, or required for a security fix, etc.
> > Configuration, data format, environment variable, changes etc can all
> > be similarly incompatible. The issue we had in HADOOP-9151 was someone
> > claimed it is not an incompatible change because it doesn't break API
> > compatibility even though it breaks wire compatibility. So let's be
> > clear about the types of incompatibility we are or are not permitting.
> > For example, will it be OK to merge a change before 2.2.0-beta that
> > requires an HDFS metadata upgrade? Or breaks client server wire
> > compatibility?  I've been assuming that changing an API annotated
> > Public/Stable still requires multiple major releases (one to deprecate
> > and one to remove), does the alpha label change that? To some people
> > the "alpha", "beta" label implies instability in terms of
> > quality/features, while to others it means unstable APIs (and to some
> > both) so it would be good to spell that out. In short, agree that we
> > really need to figure out what changes are permitted in what releases,
> > and we should update the docs accordingly (there's a start here:
> > http://wiki.apache.org/hadoop/Roadmap).
> >
> > Note that the 2.0.0 alpha release vote thread was clear that we
> > thought were all in agreement that we'd like to keep client/server
> > compatible post 2.0 - and there was no push back. We pulled a number
> > of jiras into the 2.0 release explicitly so that we could preserve
> > client/server compatibility going forward.  Here's the relevant part
> > of the thread as a refresher: http://s.apache.org/gQ
> >
> > "2) HADOOP-8285 and HADOOP-8366 changed the wire format for the RPC
> > envelope in branch-2, but didn't make it into this rc. So, that would
> > mean that future alphas would not be protocol-compatible with this
> > alpha. Per a discussion a few weeks ago, I think we all were in
> > agreement that, if possible, we'd like all 2.x to be compatible for
> > client-server communication, at least (even if we don't support
> > cross-version for the intra-cluster protocols)"
> >
> > Thanks,
> > Eli
> >
> > On Tue, Jan 29, 2013 at 12:56 PM, Arun C Murthy <acm@hortonworks.com>
> wrote:
> >> Folks,
> >>
> >> There has been some discussions about incompatible changes in the
> hadoop-2.x.x-alpha releases on HADOOP-9070, HADOOP-9151, HADOOP-9192 and
> few other jiras. Frankly, I'm surprised about some of them since the
> 'alpha' moniker was precisely to harden apis by changing them if necessary,
> borne out by the fact that every  single release in hadoop-2 chain has had
> incompatible changes. This happened since we were releasing early, moving
> fast and breaking things. Furthermore, we'll have more in future as move
> towards stability of hadoop-2 similar to HDFS-4362, HDFS-4364 et al in HDFS
> and YARN-142 (api changes) for YARN.
> >>
> >> So, rather than debate more, I had a brief chat with Suresh and Todd.
> Todd suggested calling the next release as hadoop-2.1.0-alpha to indicate
> the incompatibility a little better. This makes sense to me, as long as we
> are clear that we won't make any further *feature* releases in hadoop-2.0.x
> series (obviously we might be forced to do security/bug-fix release).
> >>
> >> Going forward, I'd like to start locking down apis/protocols for a
> 'beta' release. This way we'll have one *final* opportunity post
> hadoop-2.1.0-alpha to make incompatible changes if necessary and we can
> call it hadoop-2.2.0-beta.
> >>
> >> Post hadoop-2.2.0-beta we *should* lock down and not allow incompatible
> changes. This will allow us to go on to a hadoop-2.3.0 as a GA release.
> This forces us to do a real effort on making sure we lock down for
> hadoop-2.2.0-beta.
> >>
> >> In summary:
> >> # I plan to now release hadoop-2.1.0-alpha (this week).
> >> # We make a real effort to lock down apis/protocols and release
> hadoop-2.2.0-beta, say in March.
> >> # Post 'beta' release hadoop-2.3.0 as 'stable' sometime in May.
> >>
> >> I'll start a separate thread on 'locking protocols' w.r.t
> client-protocols v/s internal protocols (to facilitate rolling upgrades
> etc.), let's discuss this one separately.
> >>
> >> Makes sense? Thoughts?
> >>
> >> thanks,
> >> Arun
> >>
> >> PS:  Between hadoop-2.2.0-beta and hadoop-2.3.0 we *might* be forced to
> make some incompatible changes due to *unforeseen circumstances*, but no
> more gratuitous changes are allowed.
> >>
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>
>

--20cf303b4219015f0c04d49af35a--