Return-Path: X-Original-To: apmail-hadoop-hdfs-dev-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 11377176CC for ; Sat, 7 Mar 2015 01:20:12 +0000 (UTC) Received: (qmail 77950 invoked by uid 500); 7 Mar 2015 01:20:07 -0000 Delivered-To: apmail-hadoop-hdfs-dev-archive@hadoop.apache.org Received: (qmail 77850 invoked by uid 500); 7 Mar 2015 01:20:07 -0000 Mailing-List: contact hdfs-dev-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hdfs-dev@hadoop.apache.org Delivered-To: mailing list hdfs-dev@hadoop.apache.org Received: (qmail 77808 invoked by uid 99); 7 Mar 2015 01:20:07 -0000 Received: from mail-relay.apache.org (HELO mail-relay.apache.org) (140.211.11.15) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 07 Mar 2015 01:20:07 +0000 Received: from mail-we0-f180.google.com (mail-we0-f180.google.com [74.125.82.180]) by mail-relay.apache.org (ASF Mail Server at mail-relay.apache.org) with ESMTPSA id C86E81A0490; Sat, 7 Mar 2015 01:20:06 +0000 (UTC) Received: by wevm14 with SMTP id m14so62730350wev.8; Fri, 06 Mar 2015 17:20:05 -0800 (PST) MIME-Version: 1.0 X-Received: by 10.194.122.232 with SMTP id lv8mr33557808wjb.146.1425691205326; Fri, 06 Mar 2015 17:20:05 -0800 (PST) Received: by 10.28.158.17 with HTTP; Fri, 6 Mar 2015 17:20:05 -0800 (PST) In-Reply-To: <127F8CDF-EB05-4192-88AB-00146D3A257A@hortonworks.com> References: <173088578.3630972.1425591897712.JavaMail.yahoo@mail.yahoo.com> <127F8CDF-EB05-4192-88AB-00146D3A257A@hortonworks.com> Date: Fri, 6 Mar 2015 17:20:05 -0800 Message-ID: Subject: Re: Looking to a Hadoop 3 release From: Chris Douglas To: "common-dev@hadoop.apache.org" Cc: "hdfs-dev@hadoop.apache.org" , Jason Lowe , "mapreduce-dev@hadoop.apache.org" , "yarn-dev@hadoop.apache.org" Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable On Fri, Mar 6, 2015 at 4:32 PM, Vinod Kumar Vavilapalli wrote: > I'd encourage everyone to post their wish list on the Roadmap wiki that *= warrants* making incompatible changes forcing us to go 3.x. This is a useful exercise, but not a prerequisite to releasing 3.0.0 as an alpha off of trunk, right? Andrew summarized the operating assumptions for anyone working on it: rolling upgrades still work, wire compat is preserved, breaking changes may get rolled back when branch-3 is in beta (so be very conservative, notify others loudly). This applies to branches merged to trunk, also. > +1 to Jason's comments on general. We can keep rolling alphas that downst= ream can pick up, but I'd also like us to clarify the exit criterion for a = GA release of 3.0 and its relation to the life of 2.x if we are going this = route. This brings us back to the roadmap discussion, and a collective agre= ement about a logical step at a future point in time where we say we have e= nough incompatible features in 3.x that we can stop putting more of them an= d start stabilizing it. We'll have this discussion again. We don't need to reach consensus on the roadmap, just that each artifact reflects the output of the project. > Irrespective of that, here is my proposal in the interim: > - Run JDK7 + JDK8 first in a compatible manner like I mentioned before f= or atleast two releases in branch-2: say 2.8 and 2.9 before we consider tak= ing up the gauntlet on 3.0. > - Continue working on the classpath isolation effort and try making it a= s compatible as is possible for users to opt in and migrate easily. +1 for 2.x, but again I don't understand the sequencing. -C > On Mar 5, 2015, at 1:44 PM, Jason Lowe wrot= e: > >> I'm OK with a 3.0.0 release as long as we are minimizing the pain of mai= ntaining yet another release line and conscious of the incompatibilities go= ing into that release line. >> For the former, I would really rather not see a branch-3 cut so soon. I= t's yet another line onto which to cherry-pick, and I don't see why we need= to add this overhead at such an early phase. We should only create branch= -3 when there's an incompatible change that the community wants and it shou= ld _not_ go into the next major release (i.e.: it's for Hadoop 4.0). We ca= n develop 3.0 alphas and betas on trunk and release from trunk in the inter= im. IMHO we need to stop treating trunk as a place to exile patches. >> >> For the latter, I think as a community we need to evaluate the benefits = of breaking compatibility against the costs of migrating. Each time we bre= ak compatibility we create a hurdle for people to jump when they move to th= e new release, and we should make those hurdles worth their time. For exam= ple, wire-compatibility has been mentioned as part of this. Any feature th= at breaks wire compatibility better be absolutely amazing, as it creates a = huge hurdle for people to jump. >> To summarize:+1 for a community-discussed roadmap of what we're breaking= in Hadoop 3 and why it's worth it for users >> -1 for creating branch-3 now, we can release from trunk until the next i= ncompatibility for Hadoop 4 arrives >> +1 for baking classpath isolation as opt-in on 2.x and eventually defaul= t on in 3.0 >> Jason >> From: Andrew Wang >> To: "hdfs-dev@hadoop.apache.org" >> Cc: "common-dev@hadoop.apache.org" ; "mapr= educe-dev@hadoop.apache.org" ; "yarn-dev@h= adoop.apache.org" >> Sent: Wednesday, March 4, 2015 12:15 PM >> Subject: Re: Looking to a Hadoop 3 release >> >> Let's not dismiss this quite so handily. >> >> Sean, Jason, and Stack replied on HADOOP-11656 pointing out that while w= e >> could make classpath isolation opt-in via configuration, what we really >> want longer term is to have it on by default (or just always on). Stack = in >> particular points out the practical difficulties in using an opt-in meth= od >> in 2.x from a downstream project perspective. It's not pretty. >> >> The plan that both Sean and Jason propose (which I support) is to have a= n >> opt-in solution in 2.x, bake it there, then turn it on by default >> (incompatible) in a new major release. I think this lines up well with m= y >> proposal of some alphas and betas leading up to a GA 3.x. I'm also willi= ng >> to help with 2.x release management if that would help with testing this >> feature. >> >> Even setting aside classpath isolation, a new major release is still >> justified by JDK8. Somehow this is being ignored in the discussion. Alle= n, >> historically the voice of the user in our community, just highlighted it= as >> a major compatibility issue, and myself and Tucu have also expressed our >> very strong concerns about bumping this in a minor release. 2.7's bump i= s a >> unique exception, but this is not something to be cited as precedent or >> policy. >> >> Where does this resistance to a new major release stem from? As I've >> described from the beginning, this will look basically like a 2.x releas= e, >> except for the inclusion of classpath isolation by default and target >> version JDK8. I've expressed my desire to maintain API and wire >> compatibility, and we can audit the set of incompatible changes in trunk= to >> ensure this. My proposal for doing alpha and beta releases leading up to= GA >> also gives downstreams a nice amount of time for testing and validation. >> >> Regards, >> Andrew >> >> >> >> On Tue, Mar 3, 2015 at 2:32 PM, Arun Murthy wrote: >> >>> Awesome, looks like we can just do this in a compatible manner - nothin= g >>> else on the list seems like it warrants a (premature) major release. >>> >>> Thanks Vinod. >>> >>> Arun >>> >>> ________________________________________ >>> From: Vinod Kumar Vavilapalli >>> Sent: Tuesday, March 03, 2015 2:30 PM >>> To: common-dev@hadoop.apache.org >>> Cc: hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org; >>> yarn-dev@hadoop.apache.org >>> Subject: Re: Looking to a Hadoop 3 release >>> >>> I started pitching in more on that JIRA. >>> >>> To add, I think we can and should strive for doing this in a compatible >>> manner, whatever the approach. Marking and calling it incompatible befo= re >>> we see proposal/patch seems premature to me. Commented the same on JIRA= : >>> https://issues.apache.org/jira/browse/HADOOP-11656?focusedCommentId=3D1= 4345875&page=3Dcom.atlassian.jira.plugin.system.issuetabpanels:comment-tabp= anel#comment-14345875 >>> . >>> >>> Thanks >>> +Vinod >>> >>> On Mar 2, 2015, at 8:08 PM, Andrew Wang >> andrew.wang@cloudera.com>> wrote: >>> >>> Regarding classpath isolation, based on what I hear from our customers, >>> it's still a big problem (even after the MR classloader work). The late= st >>> Jackson version bump was quite painful for our downstream projects, and= the >>> HDFS client still leaks a lot of dependencies. Would welcome more >>> discussion of this on HADOOP-11656, Steve, Colin, and Haohui have alrea= dy >>> chimed in. >>> >>> >> >> >