Mailing-List: contact hdfs-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-dev@hadoop.apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Content-Type: text/plain; charset=windows-1252
Mime-Version: 1.0 (Mac OS X Mail 6.6 \(1510\))
Subject: Re: In hindsight... Re: Thinking ahead to hadoop-2.6
From: Allen Wittenauer <aw@altiscale.com>
In-Reply-To: 
 <CA+qbEUPAJzLuYKvwjBJ1p84LczKfdGLRxT28avmN2HhFwjFacQ@mail.gmail.com>
Date: Mon, 15 Sep 2014 22:53:33 -0700
Cc: Arun C Murthy <acm@hortonworks.com>,
 mapreduce-dev@hadoop.apache.org,
 yarn-dev@hadoop.apache.org,
 hdfs-dev@hadoop.apache.org
Content-Transfer-Encoding: quoted-printable
Message-Id: <0DC2356D-5EAB-4A48-80C3-99EA6C3B86D9@altiscale.com>
References: 
 <CAOScs9aEx5m_AGg_OynwA-VFG2PFpqsxHkq-Z8kfFgOObxQ4-A@mail.gmail.com>
 <F27FB0BD-FEA2-420F-A173-B6B8A2729CA6@altiscale.com>
 <CA+qbEUPAJzLuYKvwjBJ1p84LczKfdGLRxT28avmN2HhFwjFacQ@mail.gmail.com>
To: Hadoop Common <common-dev@hadoop.apache.org>


On Sep 15, 2014, at 11:17 AM, Colin McCabe <cmccabe@alumni.cmu.edu> =
wrote:

> On Mon, Sep 15, 2014 at 10:48 AM, Allen Wittenauer <aw@altiscale.com> =
wrote:
>>=20
>>        It=92s now September.  With the passage of time, I have a lot =
of doubts about this plan and where that trajectory takes us.
>>=20
>> * The list of changes that are already in branch-2 scare the crap out =
of any risk adverse person (Hello to my fellow operations people!). Not =
only are the number of changes extremely high, but in addition there are =
a lot of major, blockbuster features in what is supposed to be a minor =
release.  Combined with the fact that we=92ve had to do some micro =
releases, it seems to hint that branch-2 is getting less stable over =
time.
>=20
> I don't see what is so scary about 2.6, can you be more concrete?  It
> seems like a pretty normal release to me and most of the new features
> are optional.
>=20
> I also don't see why you think that "branch-2 is getting less stable
> over time."  Actually, I think that branch-2 has gotten more stable
> over time as people have finally gotten around to upgrading from 1.x
> or earlier, and contributed their efforts to addressing regressions in
> branch-2.
>=20

	Please re-read what I wrote above.


>> *  One of the plans talked about was rolling a 2.7 release that drops =
JDK6 and makes JDK7 the standard.  If 2.7 comes after 2.6 in October, =
date wise  makes it somewhere around January 2015.  JDK7 EOL=92s in =
April 2015.  So we=92ll have a viable JDK7 release for exactly 3 months. =
 Frankly, it is too late for us to talk about JDK7 and we need to start =
thinking about JDK8.
>>=20
>> * trunk is currently sitting at 3 years old.  There is a lot of stuff =
that has been hanging around that really needs to get into people hands =
so that we can start stabilizing it for a =93real=94 release.
>=20
> We have been pretty careful about minimizing trunk's divergence from
> branch-2.  I can't think of an example of anything in trunk that
> "really needs to get into people's hands"-- did I forget something?

	There isn't anything in *any* of the recent branch-2 releases =
that qualifies as "really needs to get into people's hands" either, so =
I'm not sure what your point here is.

>> To me this all says one thing:
>>=20
>>        Drop the 2.6.0 release, branch trunk, and start rolling a =
3.0.0-alpha with JDK8 as the minimum.  2.5.1 becomes the base for all =
sustaining work.  This gives the rest of the community time to move to =
JDK8 if they haven=92t already.  For downstream vendors, it gives a =
roadmap for their customers who will be asking about JDK8 sooner rather =
than later.  By the time 3.0 stabilizes, we=92re probably looking at =
April, which is perfect timing.
>>=20
>>        One of the issues I=92ve heard mention is that 3.0 doesn=92t =
have anything =93compelling=94 in it.  Well, dropping 2.6 makes the =
feature list the carrot, JDK8 support is obviously the stick.
>>=20
>>        Thoughts?
>=20
> As we've discussed before, supporting JDK8 is very different from
> forcing people to use JDK8.  branch-2 and Hadoop 2.6 most certainly
> should support JDK8, and most certainly NOT force people to use JDK8.

	We aren't the ones forcing the JDK8 issue: Oracle is.  Users who =
want a viable, supported JDK will be moving to 8 by April.=20

> Cloudera has been using JDK7 internally for a long time, and
> recommending it to customers too.

	Thank you for using your vendor hat to support my point:  JDK7 =
works fine with Hadoop 2 now, so those users who can't upgrade to JDK8 =
for whatever reason still have a viable option if they want to use =
Hadoop.  After all, again as you point out above, all of the new bits =
are optional features and could get pushed to a later release.  We can =
continue to make security and critical bug fixes against 2.5.x and start =
pushing those optional features into a newly viable 3.x release.  It =
also supports our major.minor.micro nomenclature as well as puts us in =
line with a lot of other Apache projects. (as pointed out to me today: =
http://t.co/Ry8Zu6yZkd )

> Some developers are using JDK8 as
> well.  It works fine (although I'm sure there will be bugs and
> workarounds that get reported and fixed as more people migrate).
=09
	All of the people that I know that are doing JDK8 are applying =
patches to make it work.  (See HADOOP-11090)

>  I don't see this particular issue as a reason to change the schedule.

	Look at this from a long-term perspective.  If we make the move =
to support JDK7 as the base line in the next few months, we're =
essentially saying that we're willing to keep JDK7 working for the next =
3-4 years (given our normal JVM update process).  This isn't a viable =
strategy given that Oracle (you know, the people that provide the JVM) =
is about to drop support/provide updates in 7 months and JDK9 in early =
access with an expected ship date in 2016.  JDK7 will be ancient at that =
point, even worse that JDK6 feels now.