Return-Path: X-Original-To: apmail-hadoop-common-dev-archive@www.apache.org Delivered-To: apmail-hadoop-common-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 99FD710A1A for ; Thu, 10 Apr 2014 04:14:56 +0000 (UTC) Received: (qmail 99150 invoked by uid 500); 10 Apr 2014 04:14:52 -0000 Delivered-To: apmail-hadoop-common-dev-archive@hadoop.apache.org Received: (qmail 98531 invoked by uid 500); 10 Apr 2014 04:14:51 -0000 Mailing-List: contact common-dev-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: common-dev@hadoop.apache.org Delivered-To: mailing list common-dev@hadoop.apache.org Received: (qmail 98521 invoked by uid 99); 10 Apr 2014 04:14:49 -0000 Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Apr 2014 04:14:49 +0000 Received: from localhost (HELO mail-ve0-f170.google.com) (127.0.0.1) (smtp-auth username vinayakumarb, mechanism plain) by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Apr 2014 04:14:49 +0000 Received: by mail-ve0-f170.google.com with SMTP id pa12so3018696veb.29 for ; Wed, 09 Apr 2014 21:14:48 -0700 (PDT) MIME-Version: 1.0 X-Received: by 10.58.185.145 with SMTP id fc17mr12804316vec.14.1397103288266; Wed, 09 Apr 2014 21:14:48 -0700 (PDT) Received: by 10.58.46.179 with HTTP; Wed, 9 Apr 2014 21:14:48 -0700 (PDT) Received: by 10.58.46.179 with HTTP; Wed, 9 Apr 2014 21:14:48 -0700 (PDT) In-Reply-To: References: <92FD1DEAAC387041B70D5237A1BE3A1D0CFC835813@MX33A.corp.emc.com> Date: Thu, 10 Apr 2014 09:44:48 +0530 Message-ID: Subject: Re: Plans of moving towards JDK7 in trunk From: Vinayakumar B To: common-dev@hadoop.apache.org Content-Type: multipart/alternative; boundary=047d7b67052f36356904f6a87544 --047d7b67052f36356904f6a87544 Content-Type: text/plain; charset=ISO-8859-1 +1 for keeping jdk 6 suppprt in branch-2 and start using jdk 7 in trunk. I agree that this approach makes patch generation difficult for branch-2 and trunk. Also the actual benefit and real issues after start using jdk7 will be known only if atleast one of the release is out in trunk version. Regards, Vinay I think this thread isn't so much about whether java7, 8 etc features are valuable, they are useful of course, and we'll want to adopt them, it's a question of how we adopt them and in which releases. For the sake of this discussion we should separate the runtime from the programming APIs. Users are already migrating to the java7 runtime for most of the reasons listed below (support, performance, bugs, etc), and the various distributions cert their Hadoop 2 based distributions on java7. This gives users many of the benefits of java7, without forcing users off java6. Ie Hadoop does not need to switch to the java7 programming APIs to make sure everyone has a supported runtime. The question here is really about when Hadoop, and the Hadoop ecosystem (since adjacent projects often end up in the same classpath) start using the java7 programming APIs and therefore break compatibility with java6 runtimes. I think our java6 runtime users would consider dropping support for their java runtime in an update of a major release to be an incompatible change (the binaries stop working on their current jvm). That may be worth it if we can articulate sufficient value to offset the cost (they have to upgrade their environment, might make rolling upgrades stop working, etc), but I've not yet heard an argument that articulates the value relative to the cost. Eg upgrading to the java7 APIs allows us to pull in dependencies with new major versions, but only if those dependencies don't break compatibility (which is likely given that our classpaths aren't so isolated), and, realistically, only if the entire Hadoop stack moves to java7 as well (eg we have to recompile HBase to generate v1.7 binaries even if they stick on API v1.6). I'm not aware of a feature, bug etc that really motivates this. An alternate approach is to keep the current stable release series (v2.x) as is, and start using new APIs in trunk (for v3). This will be a major upgrade for Hadoop and therefore an incompatible change like this is to be expected (it would be great if this came with additional changes to better isolate classpaths and dependencies from each other). It allows us to continue to support multiple types of users with different branches, vs forcing all users onto a new version. It of course means that 2.x users will not get the benefits of the new API, but its unclear what those benefits are given they can already get the benefits of adopting the newer java runtimes today. Thanks, Eli On Wed, Apr 9, 2014 at 9:38 AM, Andrew Purtell wrote: > A Java 8 runtime would also offer transparent performance improvements like > a reimplementation of ConcurrentSkipListMap, C2 support for AES cipher > acceleration with native CPU instructions, perf improvements for going from > String to byte[] or vice versa, and IIRC after 8u20 monitor lock elision > using restricted transactional memory with hardware support (if available). > Getting away from fully transparent changes but tractable to deal with > using reflection, removal of the permanent generation, support for AEAD > cipher modes like AES-GCM, stronger cipher and key exchange algorithms, TLS > 1.2, support for some krb 5 features not handled previously. > > > > On Tue, Apr 8, 2014 at 7:44 PM, Raymie Stata wrote: > >> > It might make sense to try to enumerate the benefits of switching to >> > Java7 APIs and dependencies. >> >> - Java7 introduced a huge number of language, byte-code, API, and >> tooling enhancements! Just to name a few: try-with-resources, newer >> and stronger encyrption methods, more scalable concurrency primitives. >> See http://www.slideshare.net/boulderjug/55-things-in-java-7 >> >> - We can't update current dependencies, and we can't add cool new ones. >> >> - Putting language/APIs aside, don't forget that a huge amount of effort >> goes into qualifying for Java6 (at least, I hope the folks claiming to >> support Java6 are putting in such an effort :-). Wouldn't Hadoop >> users/customers be better served if qualification effort went into >> Java7/8 versus Java6/7? >> >> Getting to Java7 as a development env (and Java8 as a runtime env) >> seems like a no-brainer. Question is: How? >> >> On Tue, Apr 8, 2014 at 10:21 AM, Sandy Ryza >> wrote: >> > It might make sense to try to enumerate the benefits of switching to >> Java7 >> > APIs and dependencies. IMO, the ones listed so far on this thread don't >> > make a compelling enough case to drop Java6 in branch-2 on any time >> frame, >> > even if this means supporting Java6 through 2015. For example, the >> change >> > in RawLocalFileSystem semantics might be an incompatible change for >> > branch-2 any way. >> > >> > >> > On Tue, Apr 8, 2014 at 10:05 AM, Karthik Kambatla > >wrote: >> > >> >> +1 to NOT breaking compatibility in branch-2. >> >> >> >> I think it is reasonable to require JDK7 for trunk, if we limit use of >> >> JDK7-only API to security fixes etc. If we make other optimizations >> (like >> >> IO), it would be a pain to backport things to branch-2. I guess this all >> >> depends on when we see ourselves shipping Hadoop-3. Any ideas on that? >> >> >> >> >> >> On Tue, Apr 8, 2014 at 9:19 AM, Eli Collins wrote: >> >> >> >> > On Tue, Apr 8, 2014 at 2:00 AM, Ottenheimer, Davi >> >> > wrote: >> >> > >> From: Eli Collins [mailto:eli@cloudera.com] >> >> > >> Sent: Monday, April 07, 2014 11:54 AM >> >> > >> >> >> > >> >> >> > >> IMO we should not drop support for Java 6 in a minor update of a >> >> stable >> >> > >> release (v2). I don't think the larger Hadoop user base would >> find it >> >> > >> acceptable that upgrading to a minor update caused their systems to >> >> stop >> >> > >> working because they didn't upgrade Java. There are people still >> >> getting >> >> > >> support for Java 6. ... >> >> > >> >> >> > >> Thanks, >> >> > >> Eli >> >> > > >> >> > > Hi Eli, >> >> > > >> >> > > Technically you are correct those with extended support get critical >> >> > security fixes for 6 until the end of 2016. I am curious whether many >> of >> >> > those are in the Hadoop user base. Do you know? My guess is the vast >> >> > majority are within Oracle's official public end of life, which was >> over >> >> 12 >> >> > months ago. Even Premier support ended Dec 2013: >> >> > > >> >> > > http://www.oracle.com/technetwork/java/eol-135779.html >> >> > > >> >> > > The end of Java 6 support carries much risk. It has to be >> considered in >> >> > terms of serious security vulnerabilities such as CVE-2013-2465 with >> CVSS >> >> > score 10.0. >> >> > > >> >> > > http://www.cvedetails.com/cve/CVE-2013-2465/ >> >> > > >> >> > > Since you mentioned "caused systems to stop" as an example of what >> >> would >> >> > be a concern to Hadoop users, please note the CVE-2013-2465 >> availability >> >> > impact: >> >> > > >> >> > > "Complete (There is a total shutdown of the affected resource. The >> >> > attacker can render the resource completely unavailable.)" >> >> > > >> >> > > This vulnerability was patched in Java 6 Update 51, but post end of >> >> > life. Apple pushed out the update specifically because of this >> >> > vulnerability (http://support.apple.com/kb/HT5717) as did some other >> >> > vendors privately, but for the majority of people using Java 6 means >> they >> >> > have a ticking time bomb. >> >> > > >> >> > > Allowing it to stay should be considered in terms of accepting the >> >> whole >> >> > risk posture. >> >> > > >> >> > >> >> > There are some who get extended support, but I suspect many just have >> >> > a if-it's-not-broke mentality when it comes to production deployments. >> >> > The current code supports both java6 and java7 and so allows these >> >> > people to remain compatible, while enabling others to upgrade to the >> >> > java7 runtime. This seems like the right compromise for a stable >> >> > release series. Again, absolutely makes sense for trunk (ie v3) to >> >> > require java7 or greater. >> >> > >> >> >> > > > > -- > Best regards, > > - Andy > > Problems worthy of attack prove their worth by hitting back. - Piet Hein > (via Tom White) --047d7b67052f36356904f6a87544--