hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Karthik Kambatla <ka...@cloudera.com>
Subject Re: Moving to JDK7, JDK8 and new major releases
Date Fri, 27 Jun 2014 22:10:40 GMT
+1 to making 2.6 the last JDK6 release.

If we want, 2.7 could be a parallel release or one soon after 2.6. We could
upgrade other dependencies that require JDK7 as well.


On Fri, Jun 27, 2014 at 3:01 PM, Arun C. Murthy <acm@hortonworks.com> wrote:

> Thanks everyone for the discussion. Looks like we have come to a pragmatic
> and progressive conclusion.
>
> In terms of execution of the consensus plan, I think a little bit of
> caution is in order.
>
> Let's give downstream projects more of a runway.
>
> I propose we inform HBase, Pig, Hive etc. that we are considering making
> 2.6 (not 2.5) the last JDK6 release and solicit their feedback. Once they
> are comfortable we can pull the trigger in 2.7.
>
> thanks,
> Arun
>
>
> > On Jun 27, 2014, at 11:34 AM, Karthik Kambatla <kasha@cloudera.com>
> wrote:
> >
> > As someone else already mentioned, we should announce one future release
> > (may be, 2.5) as the last JDK6-based release before making the move to
> JDK7.
> >
> > I am comfortable calling 2.5 the last JDK6 release.
> >
> >
> > On Fri, Jun 27, 2014 at 11:26 AM, Andrew Wang <andrew.wang@cloudera.com>
> > wrote:
> >
> >> Hi all, responding to multiple messages here,
> >>
> >> Arun, thanks for the clarification regarding MR classpaths. It sounds
> like
> >> the story there is improved and still improving.
> >>
> >> However, I think we still suffer from this at least on the HDFS side. We
> >> have a single JAR for all of HDFS, and our clients need to have all the
> fun
> >> deps like Guava on the classpath. I'm told Spark sticks a newer Guava at
> >> the front of the classpath and the HDFS client still works okay, but
> this
> >> is more happy coincidence than anything else. While we're leaking deps,
> >> we're in a scary situation.
> >>
> >> API compat to me means that an app should be able to run on a new minor
> >> version of Hadoop and not have anything break. MAPREDUCE-4421 sounds
> like
> >> it allows you to run e.g. 2.3 MR jobs on a 2.4 YARN cluster, but what
> >> should also be possible is running an HDFS 2.3 app with HDFS 2.4 JARs
> and
> >> have nothing break. If we muck with the classpath, my understanding is
> that
> >> this could break.
> >>
> >> Owen, bumping the minimum JDK version in a minor release like this
> should
> >> be a one-time exception as Tucu stated. A number of people have pointed
> out
> >> how painful a forced JDK upgrade is for end users, and it's not
> something
> >> we should be springing on them in a minor release unless we're *very*
> >> confident like in this case.
> >>
> >> Chris, thanks for bringing up the ecosystem. For CDH5, we standardized
> on
> >> JDK7 across the CDH stack, so I think that's an indication that most
> >> ecosystem projects are ready to make the jump. Is that sufficient in
> your
> >> mind?
> >>
> >> For the record, I'm also +1 on the Tucu plan. Is it too late to do this
> for
> >> 2.5? I'll offer to help out with some of the mechanics.
> >>
> >> Thanks,
> >> Andrew
> >>
> >> On Wed, Jun 25, 2014 at 4:18 PM, Chris Nauroth <
> cnauroth@hortonworks.com>
> >> wrote:
> >>
> >>> I understood the plan for avoiding JDK7-specific features in our code,
> >> and
> >>> your suggestion to add an extra Jenkins job is a great way to guard
> >> against
> >>> that.  The thing I haven't seen discussed yet is how downstream
> projects
> >>> will continue to consume our built artifacts.  If a downstream project
> >>> upgrades to pick up a bug fix, and the jar switches to 1.7 class files,
> >> but
> >>> their project is still building with 1.6, then it would be a nasty
> >>> surprise.
> >>>
> >>> These are the options I see:
> >>>
> >>> 1. Make sure all other projects upgrade first.  This doesn't sound
> >>> feasible, unless all other ecosystem projects have moved to JDK7
> already.
> >>> If not, then waiting on a single long pole project would hold up our
> >>> migration indefinitely.
> >>>
> >>> 2. We switch to JDK7, but run javac with -target 1.6 until the whole
> >>> ecosystem upgrades.  I find this undesirable, because in a certain
> sense,
> >>> it still leaves a bit of 1.6 lingering in the project.  (I'll assume
> that
> >>> end-of-life for JDK6 also means end-of-life for the 1.6 bytecode
> format.)
> >>>
> >>> 3. Just declare a clean break on some version (your earlier email said
> >> 2.5)
> >>> and start publishing artifacts built with JDK7 and no -target option.
> >>> Overall, this is my preferred option.  However, as a side effect, this
> >>> sets us up for longer-term maintenance and patch releases off of the
> 2.4
> >>> branch if a downstream project that's still on 1.6 needs to pick up a
> >>> critical bug fix.
> >>>
> >>> Of course, this is all a moot point if all the downstream ecosystem
> >>> projects have already made the switch to JDK7.  I don't know the status
> >> of
> >>> that off the top of my head.  Maybe someone else out there knows?  If
> >> not,
> >>> then I expect I can free up enough in a few weeks to volunteer for
> >> tracking
> >>> down that information.
> >>>
> >>> Chris Nauroth
> >>> Hortonworks
> >>> http://hortonworks.com/
> >>>
> >>>
> >>>
> >>> On Wed, Jun 25, 2014 at 3:12 PM, Alejandro Abdelnur <tucu@cloudera.com
> >
> >>> wrote:
> >>>
> >>>> Chris,
> >>>>
> >>>> Compiling with jdk7 and doing javac -target 1.6 is not sufficient, you
> >>> are
> >>>> still using jdk7 libraries and you could use new APIs, thus breaking
> >> jdk6
> >>>> both at compile and runtime.
> >>>>
> >>>> you need to compile with jdk6 to ensure you are not running into that
> >>>> scenario. that is why i was suggesting the nightly jdk6 build/test
> >>> jenkins
> >>>> job.
> >>>>
> >>>>
> >>>> On Wed, Jun 25, 2014 at 2:04 PM, Chris Nauroth <
> >> cnauroth@hortonworks.com
> >>>>
> >>>> wrote:
> >>>>
> >>>>> I'm also +1 for getting us to JDK7 within the 2.x line after reading
> >>> the
> >>>>> proposals and catching up on the discussion in this thread.
> >>>>>
> >>>>> Has anyone yet considered how to coordinate this change with
> >> downstream
> >>>>> projects?  Would we request downstream projects to upgrade to JDK7
> >>> first
> >>>>> before we make the move?  Would we switch to JDK7, but run javac
> >>> -target
> >>>>> 1.6 to maintain compatibility for downstream projects during an
> >> interim
> >>>>> period?
> >>>>>
> >>>>> Chris Nauroth
> >>>>> Hortonworks
> >>>>> http://hortonworks.com/
> >>>>>
> >>>>>
> >>>>>
> >>>>>> On Wed, Jun 25, 2014 at 9:48 AM, Owen O'Malley <omalley@apache.org>
> >>>>> wrote:
> >>>>>
> >>>>>> On Tue, Jun 24, 2014 at 4:44 PM, Alejandro Abdelnur <
> >>> tucu@cloudera.com
> >>>>>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> After reading this thread and thinking a bit about it, I
think it
> >>>>> should
> >>>>>> be
> >>>>>>> OK such move up to JDK7 in Hadoop
> >>>>>>
> >>>>>>
> >>>>>> I agree with Alejandro. Changing minimum JDKs is not an
> >> incompatible
> >>>>> change
> >>>>>> and is fine in the 2 branch. (Although I think it is would *not*
be
> >>>>>> appropriate for a patch release.) Of course we need to do it
with
> >>>>>> forethought and testing, but moving off of JDK 6, which is EOL'ed
> >> is
> >>> a
> >>>>> good
> >>>>>> thing. Moving to Java 8 as a minimum seems much too aggressive
and
> >> I
> >>>>> would
> >>>>>> push back on that.
> >>>>>>
> >>>>>> I'm also think that we need to let the dust settle on the Hadoop
2
> >>> line
> >>>>> for
> >>>>>> a while before we talk about Hadoop 3. It seems that it has
only
> >> been
> >>>> in
> >>>>>> the last 6 months that Hadoop 2 adoption has reached the main
> >> stream
> >>>>> users.
> >>>>>> Our user community needs time to digest the changes in Hadoop
2.x
> >>>> before
> >>>>> we
> >>>>>> fracture the community by starting to discuss Hadoop 3 releases.
> >>>>>>
> >>>>>> .. Owen
> >>>>>
> >>>>> --
> >>>>> CONFIDENTIALITY NOTICE
> >>>>> NOTICE: This message is intended for the use of the individual or
> >>> entity
> >>>> to
> >>>>> which it is addressed and may contain information that is
> >> confidential,
> >>>>> privileged and exempt from disclosure under applicable law. If the
> >>> reader
> >>>>> of this message is not the intended recipient, you are hereby
> >> notified
> >>>> that
> >>>>> any printing, copying, dissemination, distribution, disclosure or
> >>>>> forwarding of this communication is strictly prohibited. If you
have
> >>>>> received this communication in error, please contact the sender
> >>>> immediately
> >>>>> and delete it from your system. Thank You.
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> Alejandro
> >>>
> >>> --
> >>> CONFIDENTIALITY NOTICE
> >>> NOTICE: This message is intended for the use of the individual or
> entity
> >> to
> >>> which it is addressed and may contain information that is confidential,
> >>> privileged and exempt from disclosure under applicable law. If the
> reader
> >>> of this message is not the intended recipient, you are hereby notified
> >> that
> >>> any printing, copying, dissemination, distribution, disclosure or
> >>> forwarding of this communication is strictly prohibited. If you have
> >>> received this communication in error, please contact the sender
> >> immediately
> >>> and delete it from your system. Thank You.
> >>
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message