Mailing-List: contact hdfs-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-dev@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of vinodkv@hortonworks.com
 designates 64.78.52.187 as permitted sender)
Subject: Re: Looking to a Hadoop 3 release
MIME-Version: 1.0
From: Vinod Kumar Vavilapalli <vinodkv@hortonworks.com>
To: "hdfs-dev@hadoop.apache.org" <hdfs-dev@hadoop.apache.org>, Jason Lowe
	<jlowe@yahoo-inc.com>
CC: "common-dev@hadoop.apache.org" <common-dev@hadoop.apache.org>,
	"mapreduce-dev@hadoop.apache.org" <mapreduce-dev@hadoop.apache.org>,
	"yarn-dev@hadoop.apache.org" <yarn-dev@hadoop.apache.org>
Thread-Topic: Looking to a Hadoop 3 release
Thread-Index: 
 AQHQVT+BMV3VVKNk/0evUP2R3q1OgJ0Kj4sAgAAbi4CAATO/gIAAALiAgAFKgoCAAczNgIABwUYA
Date: Sat, 7 Mar 2015 00:32:58 +0000
Message-ID: <127F8CDF-EB05-4192-88AB-00146D3A257A@hortonworks.com>
References: 
 <CAGB5D2Z+tWoagDO3i6Y5kEUPygHEip2MyRrLLB0Yh7vh+MCKGA@mail.gmail.com>
 <173088578.3630972.1425591897712.JavaMail.yahoo@mail.yahoo.com>
In-Reply-To: <173088578.3630972.1425591897712.JavaMail.yahoo@mail.yahoo.com>
Accept-Language: en-US
Content-Language: en-US
Content-Type: text/plain; charset="us-ascii"
Content-ID: <BFDC1978E2728D46A7C6D152DBD30898@exch080.serverpod.net>
Content-Transfer-Encoding: quoted-printable

I'd encourage everyone to post their wish list on the Roadmap wiki that *wa=
rrants* making incompatible changes forcing us to go 3.x.

+1 to Jason's comments on general. We can keep rolling alphas that downstre=
am can pick up, but I'd also like us to clarify the exit criterion for a GA=
 release of 3.0 and its relation to the life of 2.x if we are going this ro=
ute. This brings us back to the roadmap discussion, and a collective agreem=
ent about a logical step at a future point in time where we say we have eno=
ugh incompatible features in 3.x that we can stop putting more of them and =
start stabilizing it.

Irrespective of that, here is my proposal in the interim:
 - Run JDK7 + JDK8 first in a compatible manner like I mentioned before for=
 atleast two releases in branch-2: say 2.8 and 2.9 before we consider takin=
g up the gauntlet on 3.0.
 - Continue working on the classpath isolation effort and try making it as =
compatible as is possible for users to opt in and migrate easily.

Thanks,
+Vinod

On Mar 5, 2015, at 1:44 PM, Jason Lowe <jlowe@yahoo-inc.com.INVALID> wrote:

> I'm OK with a 3.0.0 release as long as we are minimizing the pain of main=
taining yet another release line and conscious of the incompatibilities goi=
ng into that release line.
> For the former, I would really rather not see a branch-3 cut so soon.  It=
's yet another line onto which to cherry-pick, and I don't see why we need =
to add this overhead at such an early phase.  We should only create branch-=
3 when there's an incompatible change that the community wants and it shoul=
d _not_ go into the next major release (i.e.: it's for Hadoop 4.0).  We can=
 develop 3.0 alphas and betas on trunk and release from trunk in the interi=
m.  IMHO we need to stop treating trunk as a place to exile patches.
>=20
> For the latter, I think as a community we need to evaluate the benefits o=
f breaking compatibility against the costs of migrating.  Each time we brea=
k compatibility we create a hurdle for people to jump when they move to the=
 new release, and we should make those hurdles worth their time.  For examp=
le, wire-compatibility has been mentioned as part of this.  Any feature tha=
t breaks wire compatibility better be absolutely amazing, as it creates a h=
uge hurdle for people to jump.
> To summarize:+1 for a community-discussed roadmap of what we're breaking =
in Hadoop 3 and why it's worth it for users
> -1 for creating branch-3 now, we can release from trunk until the next in=
compatibility for Hadoop 4 arrives
> +1 for baking classpath isolation as opt-in on 2.x and eventually default=
 on in 3.0
> Jason
>      From: Andrew Wang <andrew.wang@cloudera.com>
> To: "hdfs-dev@hadoop.apache.org" <hdfs-dev@hadoop.apache.org>=20
> Cc: "common-dev@hadoop.apache.org" <common-dev@hadoop.apache.org>; "mapre=
duce-dev@hadoop.apache.org" <mapreduce-dev@hadoop.apache.org>; "yarn-dev@ha=
doop.apache.org" <yarn-dev@hadoop.apache.org>=20
> Sent: Wednesday, March 4, 2015 12:15 PM
> Subject: Re: Looking to a Hadoop 3 release
>=20
> Let's not dismiss this quite so handily.
>=20
> Sean, Jason, and Stack replied on HADOOP-11656 pointing out that while we
> could make classpath isolation opt-in via configuration, what we really
> want longer term is to have it on by default (or just always on). Stack i=
n
> particular points out the practical difficulties in using an opt-in metho=
d
> in 2.x from a downstream project perspective. It's not pretty.
>=20
> The plan that both Sean and Jason propose (which I support) is to have an
> opt-in solution in 2.x, bake it there, then turn it on by default
> (incompatible) in a new major release. I think this lines up well with my
> proposal of some alphas and betas leading up to a GA 3.x. I'm also willin=
g
> to help with 2.x release management if that would help with testing this
> feature.
>=20
> Even setting aside classpath isolation, a new major release is still
> justified by JDK8. Somehow this is being ignored in the discussion. Allen=
,
> historically the voice of the user in our community, just highlighted it =
as
> a major compatibility issue, and myself and Tucu have also expressed our
> very strong concerns about bumping this in a minor release. 2.7's bump is=
 a
> unique exception, but this is not something to be cited as precedent or
> policy.
>=20
> Where does this resistance to a new major release stem from? As I've
> described from the beginning, this will look basically like a 2.x release=
,
> except for the inclusion of classpath isolation by default and target
> version JDK8. I've expressed my desire to maintain API and wire
> compatibility, and we can audit the set of incompatible changes in trunk =
to
> ensure this. My proposal for doing alpha and beta releases leading up to =
GA
> also gives downstreams a nice amount of time for testing and validation.
>=20
> Regards,
> Andrew
>=20
>=20
>=20
> On Tue, Mar 3, 2015 at 2:32 PM, Arun Murthy <acm@hortonworks.com> wrote:
>=20
>> Awesome, looks like we can just do this in a compatible manner - nothing
>> else on the list seems like it warrants a (premature) major release.
>>=20
>> Thanks Vinod.
>>=20
>> Arun
>>=20
>> ________________________________________
>> From: Vinod Kumar Vavilapalli <vinodkv@hortonworks.com>
>> Sent: Tuesday, March 03, 2015 2:30 PM
>> To: common-dev@hadoop.apache.org
>> Cc: hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org;
>> yarn-dev@hadoop.apache.org
>> Subject: Re: Looking to a Hadoop 3 release
>>=20
>> I started pitching in more on that JIRA.
>>=20
>> To add, I think we can and should strive for doing this in a compatible
>> manner, whatever the approach. Marking and calling it incompatible befor=
e
>> we see proposal/patch seems premature to me. Commented the same on JIRA:
>> https://issues.apache.org/jira/browse/HADOOP-11656?focusedCommentId=3D14=
345875&page=3Dcom.atlassian.jira.plugin.system.issuetabpanels:comment-tabpa=
nel#comment-14345875
>> .
>>=20
>> Thanks
>> +Vinod
>>=20
>> On Mar 2, 2015, at 8:08 PM, Andrew Wang <andrew.wang@cloudera.com<mailto=
:
>> andrew.wang@cloudera.com>> wrote:
>>=20
>> Regarding classpath isolation, based on what I hear from our customers,
>> it's still a big problem (even after the MR classloader work). The lates=
t
>> Jackson version bump was quite painful for our downstream projects, and =
the
>> HDFS client still leaks a lot of dependencies. Would welcome more
>> discussion of this on HADOOP-11656, Steve, Colin, and Haohui have alread=
y
>> chimed in.
>>=20
>>=20
>=20
>=20