Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: general@hadoop.apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Date: Fri, 1 Mar 2013 19:52:07 -0800
From: Konstantin Boudnik <cos@apache.org>
To: general@hadoop.apache.org
Subject: Re: [DISCUSS] stabilizing Hadoop releases wrt. downstream
Message-ID: <20130302035207.GH28322@tpx>
Mail-Followup-To: general@hadoop.apache.org
References: 
 <CA+ULb+vU+i3HdA0ZmsG+N08+V7NiQtCdH678GwC3oicwfPU=Zg@mail.gmail.com>
 <CABFz-zgm8e-F+o7vX+sPbD127AA8wRzvkQhDhFVF2EAvWMRhBQ@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha256;
	protocol="application/pgp-signature"; boundary="xkXJwpr35CY/Lc3I"
Content-Disposition: inline
In-Reply-To: 
 <CABFz-zgm8e-F+o7vX+sPbD127AA8wRzvkQhDhFVF2EAvWMRhBQ@mail.gmail.com>
User-Agent: Mutt/1.5.21 (2010-09-15)

--xkXJwpr35CY/Lc3I
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

Luke,

would you mind expanding the answer a little bit? I am, for once, very curi=
ous
how test-patch process would catch something like HADOOP-9299 ?

Cos

On Wed, Feb 27, 2013 at 11:03AM, Luke Lu wrote:
> Sympathize with the sentiment...
>=20
> > The good news, is that Bigtop's charter is in big part *exactly* about
> providing you with this kind of feedback.
> > What we can NOT do is submit patches for all the issues.
>=20
> What you can do is to improve and maintain our test-patch process to do
> sanity compatibility checks for downstream projects (that you care about),
> so most breakage is caught before the patches are committed.
>=20
> I'd be happy to help reviewing the patches to improve our test patch
> process.
>=20
> __Luke
>=20
>=20
> On Tue, Feb 26, 2013 at 5:43 PM, Roman Shaposhnik <roman@shaposhnik.org>w=
rote:
>=20
> > Hi!
> >
> > for the past couple of releases of Hadoop 2.X code line the issue
> > of integration between Hadoop and its downstream projects has
> > become quite a thorny issue. The poster child here is Oozie, where
> > every release of Hadoop 2.X seems to be breaking the compatibility
> > in various unpredictable ways. At times other components (such
> > as HBase for example) also seem to be affected.
> >
> > Now, to be extremely clear -- I'm NOT talking about the *latest* version
> > of Oozie working with the *latest* version of Hadoop, instead
> > my observations come from running previous *stable*  releases
> > of Bigtop on top of Hadoop 2.X RCs.
> >
> > As many of you know Apache Bigtop aims at providing a single
> > platform for integration of Hadoop and Hadoop ecosystem projects.
> > As such we're uniquely positioned to track compatibility between
> > different Hadoop releases with regards to the downstream components
> > (things like Oozie, Pig, Hive, Mahout, etc.). Every single single RC
> > we've been pretty diligent at trying to provide integration-level feedb=
ack
> > on the quality of the upcoming release,  but it seems that our efforts
> > don't quite suffice in Hadoop 2.X stabilizing.
> >
> > Of course, one could argue that while Hadoop 2.X code line was
> > designated 'alpha' expecting much in the way of perfect integration
> > and compatibility was NOT what the Hadoop community was
> > focusing on. I can appreciate that view, but what I'm interested in
> > is the future of Hadoop 2.X not its past. Hence, here's my question
> > to all of you as a Hadoop community at large:
> >
> > Do you guys think that the project have reached a point where integrati=
on
> > and compatibility issues should be prioritized really high on the list
> > of things that make or break each future release?
> >
> > The good news, is that Bigtop's charter is in big part *exactly* about
> > providing you with this kind of feedback. We can easily tell you when
> > Hadoop behavior, with regard to downstream components, changes
> > between a previous stable release and the new RC (or even branch/trunk).
> > What we can NOT do is submit patches for all the issues. We are simply
> > too small a project and we need your help with that.
> >
> > I truly believe that we owe it to the downstream projects, and in the
> > second half of this email I will try to convince you of that.
> >
> > We all know that integration projects are impossible to pull off
> > unless there's a general consensus between all of the projects involved
> > that they indeed need to work with each other. You can NOT force
> > that notion, but you can always try to influence. This relationship
> > goes both ways.
> >
> > Consider a question in front of the downstream communities
> > of  whether or not to adopt Hadoop 2.X as the basis. To answer
> > that question each downstream project has to be reasonably
> > sure that their concerns will NOT fall on deaf ears and that
> > Hadoop developers are, essentially, 'ready' for them to pick
> > up Hadoop 2.X. I would argue that so far the Hadoop community
> > had gone out of its way to signal that 2.X codeline is NOT
> > ready for the downstream.
> >
> > I would argue that moving forward this is a really unfortunate
> > situation that may end up undermining the long term success
> > of Hadoop 2.X if we don't start addressing the problem. Think
> > about it -- 90% of unit tests that run downstream on Apache
> > infrastructure are still exercising Hadoop 1.X underneath.
> > In fact, if you were to forcefully make, lets say, HBase's
> > unit tests run on top of Hadoop 2.X quite a few of them
> > are going to fail. Hadoop community is, in effect, cutting
> > itself off from the biggest source of feedback -- its downstream
> > users. This in turn:
> >
> >   * leaves Hadoop project in a perpetual state of broken
> >     windows syndrome.
> >
> >   * leaves Apache Hadoop 2.X releases in a state considerably
> >     inferior to the releases *including* Apache Hadoop done by the
> >     vendors. The users have no choice but to alight themselves
> >     with vendor offerings if they wish to utilize latest Hadoop
> > functionality.
> >     The artifact that is know as Apache Hadoop 2.X stopped being
> >     a viable choice thus fracturing the user community and reducing
> >     the benefits of a commonly deployed codebase.
> >
> >    * leaves downstream projects of Hadoop  in a jaded state where
> >      they legitimately get very discouraged and frustrated and eventual=
ly
> >      give up thinking that -- well, we work with one release of Hadoop
> >      (the stable one Hadoop 1.X) and we shall wait for the Hadoop
> >      community to get their act together.
> >
> > In my view (shared by quite a few members of the Apache Bigtop) we
> > can definitely do better than this if we all agree that the proposed
> > first 'beta' release of Hadoop 2.0.4 is the right time for it to happen.
> >
> > It is about time Hadoop 2.X community wins back all those end users
> > and downstream projects that got left behind during the alpha
> > stabilization phase.
> >
> > Thanks,
> > Roman.
> >

--xkXJwpr35CY/Lc3I
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)

iF4EAREIAAYFAlExd2cACgkQenyFlstYjhICOQEA16mNuZCj8PK0qB/5e07luGv8
33LpXHylDmarr7S0PdoA/jRcMOLgcRd27p6I7jeQzFJhx1A+QEEJouZ8N0sPbo1c
=3Bgh
-----END PGP SIGNATURE-----

--xkXJwpr35CY/Lc3I--