hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Colin McCabe <cmcc...@alumni.cmu.edu>
Subject Re: Plans of moving towards JDK7 in trunk
Date Wed, 18 Jun 2014 17:53:41 GMT
I think we should come up with a plan for when the next Hadoop release
will drop support for JDK6.  We all know that day needs to come... the
only question is when.  I agree that writing the JDK7-only code
doesn't seem very productive unless we have a plan for when it will be
released and usable.

best,
Colin

On Tue, Jun 17, 2014 at 10:08 PM, Andrew Wang <andrew.wang@cloudera.com> wrote:
> Reviving this thread, I noticed there's been a patch and +1 on
> HADOOP-10530, and I don't think we actually reached a conclusion.
>
> I (and others) have expressed concerns about moving to JDK7 for trunk.
> Summarizing a few points:
>
> - We can't move to JDK7 in branch-2 because of compatibility
> - branch-2 is currently the only Hadoop release vehicle, there are no plans
> for a trunk-based Hadoop 3
> - Introducing JDK7-only APIs in trunk will increase divergence with
> branch-2 and make backports harder
> - Almost all developers care only about branch-2, since it is the only
> release vehicle
>
> With this in mind, I struggle to see any upsides to introducing JDK7-only
> APIs to trunk. Please let's not do anything on HADOOP-10530 or related
> until we agree on this.
>
> Thanks,
> Andrew
>
>
> On Mon, Apr 14, 2014 at 3:31 PM, Steve Loughran <stevel@hortonworks.com>
> wrote:
>
>> On 14 April 2014 17:46, Andrew Purtell <apurtell@apache.org> wrote:
>>
>> > How well is trunk tested? Does anyone deploy it with real applications
>> > running on top? When will the trunk codebase next be the basis for a
>> > production release? An impromptu diff of hadoop-common trunk against
>> > branch-2 as of today is 38,625 lines. Can they be said to be the same
>> > animal? I ask because any disincentive toward putting code in trunk is
>> > beside the point, if the only target worth pursuing today is branch-2
>> > unless one doesn't care if the code is released for production use.
>> > Questions on whither JDK6 or JDK7+ (or JRE6 versus JRE7+) only matter for
>> > the vast majority of Hadoopers if talking about branch-2.
>> >
>> >
>> I think its partly a timescale issue; its also because the 1-2 transition
>> was so significant, especially at the YARN layer, that it's still taking
>> time to trickle through.
>>
>> If you do want code to ship this year, branch-2 is where you are going to
>> try and get it in -and like you say, that's where things get tried in the
>> field. At the same time, the constraints of stability are holding us back
>> -already-.
>>
>> I don't see why we should have such another major 1-2 transition in future;
>> the rate that Arun is pushing out 2.x releases its almost back to the 0.1x
>> timescale -though at that point most people were fending for themselves and
>> expectations of stability were less. We do want smaller version increments
>> in future, which branch-2 is -mostly- delivering.
>>
>> While Java 7 doesn't have some must-have features, Java 8 is a significant
>> improvement in the language, and we should be looking ahead to that, maybe
>> even doing some leading-edge work on the side, so the same discussion
>> doesn't come up in two years time when java 7 goes EOL.
>>
>>
>> -steve
>>
>> (personal opinions only, etc, )
>>
>>
>> >
>> > On Mon, Apr 14, 2014 at 9:22 AM, Colin McCabe <cmccabe@alumni.cmu.edu
>> > >wrote:
>> >
>> > > I think the bottom line here is that as long as our stable release
>> > > uses JDK6, there is going to be a very, very strong disincentive to
>> > > put any code which can't run on JDK6 into trunk.
>> > >
>> > > Like I said earlier, the traditional reason for putting something in
>> > > trunk but not the stable release is that it needs more testing.  If a
>> > > stable release that drops support for JDK6 is more than a year away,
>> > > does it make sense to put anything in trunk like that?  What might
>> > > need more than a year of testing?  Certainly not changes to
>> > > LocalFileSystem to use the new APIs.  I also don't think an upgrade to
>> > > various libraries qualifies.
>> > >
>> > > It might be best to shelve this for now, like we've done in the past,
>> > > until we're ready to talk about a stable release that requires JDK7+.
>> > > At least that's my feeling.
>> > >
>> > > If we're really desperate for the new file APIs JDK7 provides, we
>> > > could consider using loadable modules for it in branch-2.  This is
>> > > similar to how we provide JNI versions of certain things on certain
>> > > platforms, without dropping support for the other platforms.
>> > >
>> > > best,
>> > > Colin
>> > >
>> > > On Sun, Apr 13, 2014 at 10:39 AM, Raymie Stata <rstata@altiscale.com>
>> > > wrote:
>> > > > There's an outstanding question addressed to me: "Are there
>> particular
>> > > > features or new dependencies that you would like to contribute (or
>> see
>> > > > contributed) that require using the Java 1.7 APIs?"  The question
>> > > > misses the point: We'd figure out how to write something we wanted
to
>> > > > contribute to Hadoop against the APIs of Java4 if that's what it took
>> > > > to get them into a stable release.  And at current course and speed,
>> > > > that's how ridiculous things could get.
>> > > >
>> > > > To summarize, it seems like there's a vague consensus that it might
>> be
>> > > > okay to eventually allow the use of Java7 in trunk, but there's no
>> > > > decision.  And there's been no answer to the concern that even if
>> such
>> > > > dependencies were allowed in Java7, the only people using them would
>> > > > be people who uninterested in getting their patches into a stable
>> > > > release of Hadoop on any knowable timeframe, which doesn't bode well
>> > > > for the ability to stabilize that Java7 code when it comes time to
>> > > > attempt to.
>> > > >
>> > > > I don't have more to add, so I'll go back to lurking.  It'll be
>> > > > interesting to see where we'll be standing a year from now.
>> > > >
>> > > > On Sun, Apr 13, 2014 at 2:09 AM, Tsuyoshi OZAWA
>> > > > <ozawa.tsuyoshi@gmail.com> wrote:
>> > > >> Hi,
>> > > >>
>> > > >> +1 for Karthik's idea(non-binding).
>> > > >>
>> > > >> IMO, we should keep the compatibility between JDK 6 and JDK 7
on
>> both
>> > > branch-1
>> > > >> and branch-2, because users can be using them. For future releases
>> > that
>> > > we can
>> > > >> declare breaking compatibility(e.g. 3.0.0 release), we can use
JDK 7
>> > > >> features if we
>> > > >> can get benefits. However, it can increase maintenance costs and
>> > > distributes the
>> > > >> efforts of contributions to maintain branches. Then, I think it
is
>> > > >> reasonable approach
>> > > >> that we use limited and minimum JDK-7 APIs when we have reasons
we
>> > need
>> > > to use
>> > > >> the features.
>> > > >> By the way, if we start to use JDK 7 APIs, we should declare the
>> basis
>> > > >> when to use
>> > > >> JDK 7 APIs on Wiki not to confuse contributors.
>> > > >>
>> > > >> Thanks,
>> > > >> - Tsuyoshi
>> > > >>
>> > > >> On Wed, Apr 9, 2014 at 11:44 AM, Raymie Stata <rstata@altiscale.com
>> >
>> > > wrote:
>> > > >>>> It might make sense to try to enumerate the benefits of
switching
>> to
>> > > >>>> Java7 APIs and dependencies.
>> > > >>>
>> > > >>>   - Java7 introduced a huge number of language, byte-code,
API, and
>> > > >>> tooling enhancements!  Just to name a few: try-with-resources,
>> newer
>> > > >>> and stronger encyrption methods, more scalable concurrency
>> > primitives.
>> > > >>>  See http://www.slideshare.net/boulderjug/55-things-in-java-7
>> > > >>>
>> > > >>>   - We can't update current dependencies, and we can't add
cool new
>> > > ones.
>> > > >>>
>> > > >>>   - Putting language/APIs aside, don't forget that a huge
amount of
>> > > effort
>> > > >>> goes into qualifying for Java6 (at least, I hope the folks
claiming
>> > to
>> > > >>> support Java6 are putting in such an effort :-).  Wouldn't
Hadoop
>> > > >>> users/customers be better served if qualification effort went
into
>> > > >>> Java7/8 versus Java6/7?
>> > > >>>
>> > > >>> Getting to Java7 as a development env (and Java8 as a runtime
env)
>> > > >>> seems like a no-brainer.  Question is: How?
>> > > >>>
>> > > >>> On Tue, Apr 8, 2014 at 10:21 AM, Sandy Ryza <
>> sandy.ryza@cloudera.com
>> > >
>> > > wrote:
>> > > >>>> It might make sense to try to enumerate the benefits of
switching
>> to
>> > > Java7
>> > > >>>> APIs and dependencies.  IMO, the ones listed so far on
this thread
>> > > don't
>> > > >>>> make a compelling enough case to drop Java6 in branch-2
on any
>> time
>> > > frame,
>> > > >>>> even if this means supporting Java6 through 2015.  For
example,
>> the
>> > > change
>> > > >>>> in RawLocalFileSystem semantics might be an incompatible
change
>> for
>> > > >>>> branch-2 any way.
>> > > >>>>
>> > > >>>>
>> > > >>>> On Tue, Apr 8, 2014 at 10:05 AM, Karthik Kambatla <
>> > kasha@cloudera.com
>> > > >wrote:
>> > > >>>>
>> > > >>>>> +1 to NOT breaking compatibility in branch-2.
>> > > >>>>>
>> > > >>>>> I think it is reasonable to require JDK7 for trunk,
if we limit
>> use
>> > > of
>> > > >>>>> JDK7-only API to security fixes etc. If we make other
>> optimizations
>> > > (like
>> > > >>>>> IO), it would be a pain to backport things to branch-2.
I guess
>> > this
>> > > all
>> > > >>>>> depends on when we see ourselves shipping Hadoop-3.
Any ideas on
>> > > that?
>> > > >>>>>
>> > > >>>>>
>> > > >>>>> On Tue, Apr 8, 2014 at 9:19 AM, Eli Collins <eli@cloudera.com>
>> > > wrote:
>> > > >>>>>
>> > > >>>>> > On Tue, Apr 8, 2014 at 2:00 AM, Ottenheimer,
Davi
>> > > >>>>> > <Davi.Ottenheimer@emc.com> wrote:
>> > > >>>>> > >> From: Eli Collins [mailto:eli@cloudera.com]
>> > > >>>>> > >> Sent: Monday, April 07, 2014 11:54 AM
>> > > >>>>> > >>
>> > > >>>>> > >>
>> > > >>>>> > >> IMO we should not drop support for Java
6 in a minor update
>> > of a
>> > > >>>>> stable
>> > > >>>>> > >> release (v2).  I don't think the larger
Hadoop user base
>> would
>> > > find it
>> > > >>>>> > >> acceptable that upgrading to a minor
update caused their
>> > > systems to
>> > > >>>>> stop
>> > > >>>>> > >> working because they didn't upgrade
Java. There are people
>> > still
>> > > >>>>> getting
>> > > >>>>> > >> support for Java 6. ...
>> > > >>>>> > >>
>> > > >>>>> > >> Thanks,
>> > > >>>>> > >> Eli
>> > > >>>>> > >
>> > > >>>>> > > Hi Eli,
>> > > >>>>> > >
>> > > >>>>> > > Technically you are correct those with extended
support get
>> > > critical
>> > > >>>>> > security fixes for 6 until the end of 2016. I
am curious
>> whether
>> > > many of
>> > > >>>>> > those are in the Hadoop user base. Do you know?
My guess is the
>> > > vast
>> > > >>>>> > majority are within Oracle's official public
end of life, which
>> > > was over
>> > > >>>>> 12
>> > > >>>>> > months ago. Even Premier support ended Dec 2013:
>> > > >>>>> > >
>> > > >>>>> > > http://www.oracle.com/technetwork/java/eol-135779.html
>> > > >>>>> > >
>> > > >>>>> > > The end of Java 6 support carries much risk.
It has to be
>> > > considered in
>> > > >>>>> > terms of serious security vulnerabilities such
as CVE-2013-2465
>> > > with CVSS
>> > > >>>>> > score 10.0.
>> > > >>>>> > >
>> > > >>>>> > > http://www.cvedetails.com/cve/CVE-2013-2465/
>> > > >>>>> > >
>> > > >>>>> > > Since you mentioned "caused systems to stop"
as an example of
>> > > what
>> > > >>>>> would
>> > > >>>>> > be a concern to Hadoop users, please note the
CVE-2013-2465
>> > > availability
>> > > >>>>> > impact:
>> > > >>>>> > >
>> > > >>>>> > > "Complete (There is a total shutdown of
the affected
>> resource.
>> > > The
>> > > >>>>> > attacker can render the resource completely unavailable.)"
>> > > >>>>> > >
>> > > >>>>> > > This vulnerability was patched in Java 6
Update 51, but post
>> > end
>> > > of
>> > > >>>>> > life. Apple pushed out the update specifically
because of this
>> > > >>>>> > vulnerability (http://support.apple.com/kb/HT5717)
as did some
>> > > other
>> > > >>>>> > vendors privately, but for the majority of people
using Java 6
>> > > means they
>> > > >>>>> > have a ticking time bomb.
>> > > >>>>> > >
>> > > >>>>> > > Allowing it to stay should be considered
in terms of
>> accepting
>> > > the
>> > > >>>>> whole
>> > > >>>>> > risk posture.
>> > > >>>>> > >
>> > > >>>>> >
>> > > >>>>> > There are some who get extended support, but
I suspect many
>> just
>> > > have
>> > > >>>>> > a if-it's-not-broke mentality when it comes to
production
>> > > deployments.
>> > > >>>>> > The current code supports both java6 and java7
and so allows
>> > these
>> > > >>>>> > people to remain compatible, while enabling others
to upgrade
>> to
>> > > the
>> > > >>>>> > java7 runtime. This seems like the right compromise
for a
>> stable
>> > > >>>>> > release series. Again, absolutely makes sense
for trunk (ie v3)
>> > to
>> > > >>>>> > require java7 or greater.
>> > > >>>>> >
>> > > >>>>>
>> > > >>
>> > > >>
>> > > >>
>> > > >> --
>> > > >> - Tsuyoshi
>> > >
>> >
>> >
>> >
>> > --
>> > Best regards,
>> >
>> >    - Andy
>> >
>> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
>> > (via Tom White)
>> >
>>
>> --
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity to
>> which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>>

Mime
View raw message