Mailing-List: contact common-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-dev@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of andrew.wang@cloudera.com
 designates 209.85.128.176 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CA+4kjVsUJ_Ru88JpnLOch-JqGUSrSeH7Y89aLS+qvvz=nbo4qA@mail.gmail.com>
References: 
 <CABbyifM02nMVaRpARPqnSeJ8gb5x9WTTy8c18tAfr6Z4tQLy2g@mail.gmail.com>
 <CAJs-t7O2V7EuNeMt0hJN2T8aEUJHgnfTDUaBkJVUFui3ca=Npg@mail.gmail.com>
 <CABbyifOGA=c1Ytgx6=iAr-22W7zSXa1DdV1jTVX5BHg+gXe61w@mail.gmail.com>
 <CA+qbEUMJtkmfoAY1xfWKkKF-Y0FkR6_dybyN5E0+=amYA7WDcg@mail.gmail.com>
 <CA+4kjVuaByNWshmiYFcA64dvcR3TvxKaUNyJdTnAw6yuqdpdCA@mail.gmail.com>
 <CAO0ccOdEnBH_WFWnE9K=Dg7XYa4Ar0yF7nZe8dG1U6_Em8hNfg@mail.gmail.com>
 <CAPn_vTuHpDrWySH-i1-FxGbYg7d8m-yw14p2T7_ik=ZWa5Keag@mail.gmail.com>
 <92FD1DEAAC387041B70D5237A1BE3A1D0CFC835813@MX33A.corp.emc.com>
 <CAPn_vTs8-mERm2HNO=ZRH9acp=SS-Ax_yh=1SpCPGidNLptkhg@mail.gmail.com>
 <CALwhT95HxQ-n83yvM8FdTnoH4zzaEnu8HkpY2ko98k65H7jb4Q@mail.gmail.com>
 <CACBYxKKBv4e1P=YSOX1oEJYRx9yEiDs6Z0zWGSWfEuM3TFohiQ@mail.gmail.com>
 <CAO0ccOcB3UU9UPOS5T=pi-Yr8PO4iNAh76rBFq7JSnOCCwad0Q@mail.gmail.com>
 <CAAD07O+GwuybeCoLZc3fHbfWE4N3_QQPj3cwmpQ4=1RiAhF0kg@mail.gmail.com>
 <CAO0ccOc0VronqLfYfSVBDZ5M8rFZVW2Buc=P5mBBn4u6pDLG-A@mail.gmail.com>
 <CA+qbEUPXoK_hJRKVpEy_dFLdDkeTOtSz=1WBqkzkOjsdVK=YEQ@mail.gmail.com>
 <CA+RK=_BRi2pKN2VDFhsHa4aSz6ZHSSL1zr9dDF0gAxVSsZ5opQ@mail.gmail.com>
 <CA+4kjVsUJ_Ru88JpnLOch-JqGUSrSeH7Y89aLS+qvvz=nbo4qA@mail.gmail.com>
From: Andrew Wang <andrew.wang@cloudera.com>
Date: Tue, 17 Jun 2014 22:08:20 -0700
Message-ID: 
 <CAGB5D2Zr8fTvXW-3osLUjmhL822Ur9f37nXqXTNXTpwRDGKC1Q@mail.gmail.com>
Subject: Re: Plans of moving towards JDK7 in trunk
To: "common-dev@hadoop.apache.org" <common-dev@hadoop.apache.org>
Content-Type: multipart/alternative; boundary=089e0149c594e9c46704fc154097

--089e0149c594e9c46704fc154097
Content-Type: text/plain; charset=UTF-8

Reviving this thread, I noticed there's been a patch and +1 on
HADOOP-10530, and I don't think we actually reached a conclusion.

I (and others) have expressed concerns about moving to JDK7 for trunk.
Summarizing a few points:

- We can't move to JDK7 in branch-2 because of compatibility
- branch-2 is currently the only Hadoop release vehicle, there are no plans
for a trunk-based Hadoop 3
- Introducing JDK7-only APIs in trunk will increase divergence with
branch-2 and make backports harder
- Almost all developers care only about branch-2, since it is the only
release vehicle

With this in mind, I struggle to see any upsides to introducing JDK7-only
APIs to trunk. Please let's not do anything on HADOOP-10530 or related
until we agree on this.

Thanks,
Andrew


On Mon, Apr 14, 2014 at 3:31 PM, Steve Loughran <stevel@hortonworks.com>
wrote:

> On 14 April 2014 17:46, Andrew Purtell <apurtell@apache.org> wrote:
>
> > How well is trunk tested? Does anyone deploy it with real applications
> > running on top? When will the trunk codebase next be the basis for a
> > production release? An impromptu diff of hadoop-common trunk against
> > branch-2 as of today is 38,625 lines. Can they be said to be the same
> > animal? I ask because any disincentive toward putting code in trunk is
> > beside the point, if the only target worth pursuing today is branch-2
> > unless one doesn't care if the code is released for production use.
> > Questions on whither JDK6 or JDK7+ (or JRE6 versus JRE7+) only matter for
> > the vast majority of Hadoopers if talking about branch-2.
> >
> >
> I think its partly a timescale issue; its also because the 1-2 transition
> was so significant, especially at the YARN layer, that it's still taking
> time to trickle through.
>
> If you do want code to ship this year, branch-2 is where you are going to
> try and get it in -and like you say, that's where things get tried in the
> field. At the same time, the constraints of stability are holding us back
> -already-.
>
> I don't see why we should have such another major 1-2 transition in future;
> the rate that Arun is pushing out 2.x releases its almost back to the 0.1x
> timescale -though at that point most people were fending for themselves and
> expectations of stability were less. We do want smaller version increments
> in future, which branch-2 is -mostly- delivering.
>
> While Java 7 doesn't have some must-have features, Java 8 is a significant
> improvement in the language, and we should be looking ahead to that, maybe
> even doing some leading-edge work on the side, so the same discussion
> doesn't come up in two years time when java 7 goes EOL.
>
>
> -steve
>
> (personal opinions only, etc, )
>
>
> >
> > On Mon, Apr 14, 2014 at 9:22 AM, Colin McCabe <cmccabe@alumni.cmu.edu
> > >wrote:
> >
> > > I think the bottom line here is that as long as our stable release
> > > uses JDK6, there is going to be a very, very strong disincentive to
> > > put any code which can't run on JDK6 into trunk.
> > >
> > > Like I said earlier, the traditional reason for putting something in
> > > trunk but not the stable release is that it needs more testing.  If a
> > > stable release that drops support for JDK6 is more than a year away,
> > > does it make sense to put anything in trunk like that?  What might
> > > need more than a year of testing?  Certainly not changes to
> > > LocalFileSystem to use the new APIs.  I also don't think an upgrade to
> > > various libraries qualifies.
> > >
> > > It might be best to shelve this for now, like we've done in the past,
> > > until we're ready to talk about a stable release that requires JDK7+.
> > > At least that's my feeling.
> > >
> > > If we're really desperate for the new file APIs JDK7 provides, we
> > > could consider using loadable modules for it in branch-2.  This is
> > > similar to how we provide JNI versions of certain things on certain
> > > platforms, without dropping support for the other platforms.
> > >
> > > best,
> > > Colin
> > >
> > > On Sun, Apr 13, 2014 at 10:39 AM, Raymie Stata <rstata@altiscale.com>
> > > wrote:
> > > > There's an outstanding question addressed to me: "Are there
> particular
> > > > features or new dependencies that you would like to contribute (or
> see
> > > > contributed) that require using the Java 1.7 APIs?"  The question
> > > > misses the point: We'd figure out how to write something we wanted to
> > > > contribute to Hadoop against the APIs of Java4 if that's what it took
> > > > to get them into a stable release.  And at current course and speed,
> > > > that's how ridiculous things could get.
> > > >
> > > > To summarize, it seems like there's a vague consensus that it might
> be
> > > > okay to eventually allow the use of Java7 in trunk, but there's no
> > > > decision.  And there's been no answer to the concern that even if
> such
> > > > dependencies were allowed in Java7, the only people using them would
> > > > be people who uninterested in getting their patches into a stable
> > > > release of Hadoop on any knowable timeframe, which doesn't bode well
> > > > for the ability to stabilize that Java7 code when it comes time to
> > > > attempt to.
> > > >
> > > > I don't have more to add, so I'll go back to lurking.  It'll be
> > > > interesting to see where we'll be standing a year from now.
> > > >
> > > > On Sun, Apr 13, 2014 at 2:09 AM, Tsuyoshi OZAWA
> > > > <ozawa.tsuyoshi@gmail.com> wrote:
> > > >> Hi,
> > > >>
> > > >> +1 for Karthik's idea(non-binding).
> > > >>
> > > >> IMO, we should keep the compatibility between JDK 6 and JDK 7 on
> both
> > > branch-1
> > > >> and branch-2, because users can be using them. For future releases
> > that
> > > we can
> > > >> declare breaking compatibility(e.g. 3.0.0 release), we can use JDK 7
> > > >> features if we
> > > >> can get benefits. However, it can increase maintenance costs and
> > > distributes the
> > > >> efforts of contributions to maintain branches. Then, I think it is
> > > >> reasonable approach
> > > >> that we use limited and minimum JDK-7 APIs when we have reasons we
> > need
> > > to use
> > > >> the features.
> > > >> By the way, if we start to use JDK 7 APIs, we should declare the
> basis
> > > >> when to use
> > > >> JDK 7 APIs on Wiki not to confuse contributors.
> > > >>
> > > >> Thanks,
> > > >> - Tsuyoshi
> > > >>
> > > >> On Wed, Apr 9, 2014 at 11:44 AM, Raymie Stata <rstata@altiscale.com
> >
> > > wrote:
> > > >>>> It might make sense to try to enumerate the benefits of switching
> to
> > > >>>> Java7 APIs and dependencies.
> > > >>>
> > > >>>   - Java7 introduced a huge number of language, byte-code, API, and
> > > >>> tooling enhancements!  Just to name a few: try-with-resources,
> newer
> > > >>> and stronger encyrption methods, more scalable concurrency
> > primitives.
> > > >>>  See http://www.slideshare.net/boulderjug/55-things-in-java-7
> > > >>>
> > > >>>   - We can't update current dependencies, and we can't add cool new
> > > ones.
> > > >>>
> > > >>>   - Putting language/APIs aside, don't forget that a huge amount of
> > > effort
> > > >>> goes into qualifying for Java6 (at least, I hope the folks claiming
> > to
> > > >>> support Java6 are putting in such an effort :-).  Wouldn't Hadoop
> > > >>> users/customers be better served if qualification effort went into
> > > >>> Java7/8 versus Java6/7?
> > > >>>
> > > >>> Getting to Java7 as a development env (and Java8 as a runtime env)
> > > >>> seems like a no-brainer.  Question is: How?
> > > >>>
> > > >>> On Tue, Apr 8, 2014 at 10:21 AM, Sandy Ryza <
> sandy.ryza@cloudera.com
> > >
> > > wrote:
> > > >>>> It might make sense to try to enumerate the benefits of switching
> to
> > > Java7
> > > >>>> APIs and dependencies.  IMO, the ones listed so far on this thread
> > > don't
> > > >>>> make a compelling enough case to drop Java6 in branch-2 on any
> time
> > > frame,
> > > >>>> even if this means supporting Java6 through 2015.  For example,
> the
> > > change
> > > >>>> in RawLocalFileSystem semantics might be an incompatible change
> for
> > > >>>> branch-2 any way.
> > > >>>>
> > > >>>>
> > > >>>> On Tue, Apr 8, 2014 at 10:05 AM, Karthik Kambatla <
> > kasha@cloudera.com
> > > >wrote:
> > > >>>>
> > > >>>>> +1 to NOT breaking compatibility in branch-2.
> > > >>>>>
> > > >>>>> I think it is reasonable to require JDK7 for trunk, if we limit
> use
> > > of
> > > >>>>> JDK7-only API to security fixes etc. If we make other
> optimizations
> > > (like
> > > >>>>> IO), it would be a pain to backport things to branch-2. I guess
> > this
> > > all
> > > >>>>> depends on when we see ourselves shipping Hadoop-3. Any ideas on
> > > that?
> > > >>>>>
> > > >>>>>
> > > >>>>> On Tue, Apr 8, 2014 at 9:19 AM, Eli Collins <eli@cloudera.com>
> > > wrote:
> > > >>>>>
> > > >>>>> > On Tue, Apr 8, 2014 at 2:00 AM, Ottenheimer, Davi
> > > >>>>> > <Davi.Ottenheimer@emc.com> wrote:
> > > >>>>> > >> From: Eli Collins [mailto:eli@cloudera.com]
> > > >>>>> > >> Sent: Monday, April 07, 2014 11:54 AM
> > > >>>>> > >>
> > > >>>>> > >>
> > > >>>>> > >> IMO we should not drop support for Java 6 in a minor update
> > of a
> > > >>>>> stable
> > > >>>>> > >> release (v2).  I don't think the larger Hadoop user base
> would
> > > find it
> > > >>>>> > >> acceptable that upgrading to a minor update caused their
> > > systems to
> > > >>>>> stop
> > > >>>>> > >> working because they didn't upgrade Java. There are people
> > still
> > > >>>>> getting
> > > >>>>> > >> support for Java 6. ...
> > > >>>>> > >>
> > > >>>>> > >> Thanks,
> > > >>>>> > >> Eli
> > > >>>>> > >
> > > >>>>> > > Hi Eli,
> > > >>>>> > >
> > > >>>>> > > Technically you are correct those with extended support get
> > > critical
> > > >>>>> > security fixes for 6 until the end of 2016. I am curious
> whether
> > > many of
> > > >>>>> > those are in the Hadoop user base. Do you know? My guess is the
> > > vast
> > > >>>>> > majority are within Oracle's official public end of life, which
> > > was over
> > > >>>>> 12
> > > >>>>> > months ago. Even Premier support ended Dec 2013:
> > > >>>>> > >
> > > >>>>> > > http://www.oracle.com/technetwork/java/eol-135779.html
> > > >>>>> > >
> > > >>>>> > > The end of Java 6 support carries much risk. It has to be
> > > considered in
> > > >>>>> > terms of serious security vulnerabilities such as CVE-2013-2465
> > > with CVSS
> > > >>>>> > score 10.0.
> > > >>>>> > >
> > > >>>>> > > http://www.cvedetails.com/cve/CVE-2013-2465/
> > > >>>>> > >
> > > >>>>> > > Since you mentioned "caused systems to stop" as an example of
> > > what
> > > >>>>> would
> > > >>>>> > be a concern to Hadoop users, please note the CVE-2013-2465
> > > availability
> > > >>>>> > impact:
> > > >>>>> > >
> > > >>>>> > > "Complete (There is a total shutdown of the affected
> resource.
> > > The
> > > >>>>> > attacker can render the resource completely unavailable.)"
> > > >>>>> > >
> > > >>>>> > > This vulnerability was patched in Java 6 Update 51, but post
> > end
> > > of
> > > >>>>> > life. Apple pushed out the update specifically because of this
> > > >>>>> > vulnerability (http://support.apple.com/kb/HT5717) as did some
> > > other
> > > >>>>> > vendors privately, but for the majority of people using Java 6
> > > means they
> > > >>>>> > have a ticking time bomb.
> > > >>>>> > >
> > > >>>>> > > Allowing it to stay should be considered in terms of
> accepting
> > > the
> > > >>>>> whole
> > > >>>>> > risk posture.
> > > >>>>> > >
> > > >>>>> >
> > > >>>>> > There are some who get extended support, but I suspect many
> just
> > > have
> > > >>>>> > a if-it's-not-broke mentality when it comes to production
> > > deployments.
> > > >>>>> > The current code supports both java6 and java7 and so allows
> > these
> > > >>>>> > people to remain compatible, while enabling others to upgrade
> to
> > > the
> > > >>>>> > java7 runtime. This seems like the right compromise for a
> stable
> > > >>>>> > release series. Again, absolutely makes sense for trunk (ie v3)
> > to
> > > >>>>> > require java7 or greater.
> > > >>>>> >
> > > >>>>>
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> - Tsuyoshi
> > >
> >
> >
> >
> > --
> > Best regards,
> >
> >    - Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
> > (via Tom White)
> >
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

--089e0149c594e9c46704fc154097--