Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: general@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of shv.hadoop@gmail.com designates
 209.85.217.169 as permitted sender)
MIME-Version: 1.0
Date: Wed, 24 Apr 2013 20:17:36 -0700
Message-ID: 
 <CAKtuutG1m3m-3SfqsCGwsFAPUR4v5Jf23Nw0rQ4DrgiKwRSQeQ@mail.gmail.com>
Subject: Streamlining the Hadoop release process
From: Konstantin Shvachko <shv.hadoop@gmail.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Content-Type: multipart/alternative; boundary=001a11c264fe3992c404db26dc08

--001a11c264fe3992c404db26dc08
Content-Type: text/plain; charset=ISO-8859-1

 Hi everybody,

There was and is a number of discussions about Hadoop version
compatibility, feature porting, stability. I think that many problems of
Hadoop are the result of our flawed release processes and can be solved by
streamlining the releases.

It is a fact that current trunk turned into a junk-yard of partly
implemented ideas at different stages of the development. Saying this not
because trunk should not evolve, but mostly because there are no any plans
on the horizon to release anything from trunk and therefore nobody cares.

Thus now in order for A feature to make into A release the former should be
ported into an earlier branch. And that becomes a controversy because

a) most major features contain incompatible changes, and
b) they introduce massive changes, which break reliability

This destabilizes the release canceling former efforts to fix bugs and
provide working environment for the upstream projects. I mean stabilizing
and adding features are mutually exclusive activities. This is in part why
Hadoop 2 stabilization effort is perpetual.

The solution imho is to release instead of backporting. That is, produce a
new release for every or a few new major feature. Say, snapshots would be a
new release, Windows support another, InodeIDs or local reads optimization
would have been the next, etc. As the community we should release even if
that particular release is not planned to be stable, which we do anyway.

Stabilization requires meticulous work and rather stable code base. This
was done for Hadoop 1 and Hadoop 0.23 as the most recent examples.
In fact we cannot predict in advance which branch will become stable until
it is somewhere in production, which is beyond the scope of control of this
community.

My practical suggestions are:
1. Produce a series of feature releases to catch up branch-2 with trunk.
   We can prioritize features in general or let the release manager to
decide which feature to pick up from trunk.
   Version numbering is also up for discussion. I would call them 2.x and
reserve the minor numbers for subsequent stabilization bug-fix-releases.
2. Build new features in dev-branches until they are done. We do it now,
but should enforce more.
3. After branch-2 is caught up with trunk release from trunk by merging a
few new dev-branches. Releasing with 2-3 new features should be the golden
rule.
   I would call these releases 3.x
4. Disallow backporting features that have not been released from trunk.
   This is VERY important for forward going release process.
5. I'd like to propose to move the latest stable version from 1.0.X to
0.23.7.
   I think that running a release in production for 9 months on 40K nodes
is consistent with the definition of stable.

Some ideas for discussion.
Thanks,
--Konstantin

P.S. This is not to preempt discussion on stabilizing 2.0.5 started by
Roman. I am just not sure why we call it stabilization and 2.0.5 and betta,
if incompatible and new features has already been committed to the branch.
BTW as an illustration to my observations above.

--001a11c264fe3992c404db26dc08--