Return-Path: X-Original-To: apmail-hadoop-yarn-dev-archive@minotaur.apache.org Delivered-To: apmail-hadoop-yarn-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 4CE96171D7 for ; Tue, 10 Mar 2015 18:56:00 +0000 (UTC) Received: (qmail 63736 invoked by uid 500); 10 Mar 2015 18:55:50 -0000 Delivered-To: apmail-hadoop-yarn-dev-archive@hadoop.apache.org Received: (qmail 63654 invoked by uid 500); 10 Mar 2015 18:55:50 -0000 Mailing-List: contact yarn-dev-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: yarn-dev@hadoop.apache.org Delivered-To: mailing list yarn-dev@hadoop.apache.org Received: (qmail 63611 invoked by uid 99); 10 Mar 2015 18:55:50 -0000 Received: from mail-relay.apache.org (HELO mail-relay.apache.org) (140.211.11.15) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Mar 2015 18:55:50 +0000 Received: from mail-wi0-f171.google.com (mail-wi0-f171.google.com [209.85.212.171]) by mail-relay.apache.org (ASF Mail Server at mail-relay.apache.org) with ESMTPSA id 786221A02E4; Tue, 10 Mar 2015 18:55:49 +0000 (UTC) Received: by wiwl15 with SMTP id l15so5621449wiw.0; Tue, 10 Mar 2015 11:55:48 -0700 (PDT) MIME-Version: 1.0 X-Received: by 10.180.207.6 with SMTP id ls6mr117573864wic.13.1426013748223; Tue, 10 Mar 2015 11:55:48 -0700 (PDT) Received: by 10.194.189.137 with HTTP; Tue, 10 Mar 2015 11:55:48 -0700 (PDT) In-Reply-To: <1425941696034.6386@hortonworks.com> References: <1425941696034.6386@hortonworks.com> Date: Tue, 10 Mar 2015 11:55:48 -0700 Message-ID: Subject: Re: Hadoop 3.x: what about shipping trunk as a 2.x release in 2015? From: "Colin P. McCabe" To: Hadoop Common Cc: "hdfs-dev@hadoop.apache.org" , "mapreduce-dev@hadoop.apache.org" , "yarn-dev@hadoop.apache.org" Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hi Arun, Not all changes which are incompatible can be "fixed"-- sometimes an incompatibility is a necessary part of a change. For example, taking a really old library dependency with known security issues off the CLASSPATH will create incompatibilities, but it's also necessary. A minimum JDK version bump also falls in that category. There are also cases where we need to drop support for really obsolete and baroque features from the past. For example, it would be nice if we could finally get rid of the code to read pre-transactional edit logs. It's a substantial amount of code. We could argue that we should just support legacy stuff forever, but code quality will suffer. These changes need to be made sooner or later, and a major version bump is an ideal place to make them. I think that making these changes in a 2.x release is hostile to operators, as Alan commented. That's what we're trying to avoid by discussing Hadoop 3.x. Colin On Mon, Mar 9, 2015 at 3:54 PM, Arun Murthy wrote: > Colin, > > Do you have a list of incompatible changes other than the shell-script r= ewrite? If we do have others we'd have to fix them anyway for the current p= lan on hadoop-3.x right? So, I don't see the difference? > > Arun > > ________________________________________ > From: Colin P. McCabe > Sent: Monday, March 09, 2015 3:05 PM > To: hdfs-dev@hadoop.apache.org > Cc: mapreduce-dev@hadoop.apache.org; common-dev@hadoop.apache.org; yarn-d= ev@hadoop.apache.org > Subject: Re: Hadoop 3.x: what about shipping trunk as a 2.x release in 20= 15? > > Java 7 will be end-of-lifed in April 2015. I think it would be unwise > to plan a new Hadoop release against a version of Java that is almost > obsolete and (soon) no longer receiving security updates. I think > people will be willing to roll out a new version of Java for Hadoop > 3.x. > > Similarly, the whole point of bumping the major version number is the > ability to make incompatible changes. There are already a bunch of > incompatible changes in the trunk branch. Are you proposing to revert > those? Or push them into newly created feature branches? This > doesn't seem like a good idea to me. > > I would be in favor of backporting targetted incompatible changes from > trunk to branch-2. For example, we could consider pulling in Allen's > shell script rewrite. But pulling in all of trunk seems like a bad > idea at this point, if we want a 2.x release. > > best, > Colin > > On Mon, Mar 9, 2015 at 2:15 PM, Steve Loughran w= rote: >> >> If 3.x is going to be Java 8 & not backwards compatible, I don't expect = anyone wanting to use this in production until some time deep into 2016. >> >> Issue: JDK 8 vs 7 >> >> It will require Hadoop clusters to move up to Java 8. While there's dev = pull for this, there's ops pull against this: people are still in the movin= g-off Java 6 phase due to that "it's working, don't update it" philosophy. = Java 8 is compelling to us coders, but that doesn't mean ops want it. >> >> You can run JDK-8 code in a YARN cluster running on Hadoop 2.7 *today*, = the main thing is setting up JAVA_HOME. That's something we could make easi= er somehow (maybe some min Java version field in resource requests that wil= l let apps say java 8, java 9, ...). YARN could not only set up JVM paths, = it could fail-fast if a Java version wasn't available. >> >> What we can't do in hadoop coretoday is set javac.version=3D1.8 & use j= ava 8 code. Downstream code ca do that (Hive, etc); they just need to accep= t that they don't get to play on JDK7 clusters if they embrace l-expression= s. >> >> So...we need to stay on java 7 for some time due to ops pull; downstream= apps get to choose what they want. We can/could enhance YARN to make JVM c= hoice more declarative. >> >> Issue: Incompatible changes >> >> Without knowing what is proposed for "an incompatible classpath change",= I can't say whether this is something that could be made optional. If it i= sn't, then it is a python-3 class option, "rewrite your code" event, which = is going to be particularly traumatic to things like Hive that already do c= omplex CP games. I'm currently against any mandatory change here, though wo= uld love to see an optional one. And if optional, it ceases to become an in= compatible change... >> >> Issue: Getting trunk out the door >> >> The main diff from branch-2 and trunk is currently the bash script chang= es. These don't break client apps. May or may not break bigtop & other down= stream hadoop stacks, but developers don't need to worry about this: no re= compilation necessary >> >> Proposed: ship trunk as a 2.x release, compatible with JDK7 & Java code. >> >> It seems to me that I could go >> >> git checkout trunk >> mvn versions:set -DnewVersion=3D2.8.0-SNAPSHOT >> >> We'd then have a version of Hadoop-trunk we could ship later this year, = compatible at the JDK and API level with the existing java code & JDK7+ clu= sters. >> >> A classpath fix that is optional/compatible can then go out on the 2.x l= ine, saving the 3.x tag for something that really breaks things, forces all= downstream apps to set up new hadoop profiles, have separate modules & gen= erally hate the hadoop dev team >> >> This lets us tick off the "recent trunk release" and "fixed shell script= s" items, pushing out those benefits to people sooner rather than later, an= d puts off the "Hello, we've just broken your code" event for another 12+ m= onths. >> >> Comments? >> >> -Steve >> >> >>