hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Wang <andrew.w...@cloudera.com>
Subject Re: Moving to JDK7, JDK8 and new major releases
Date Fri, 27 Jun 2014 18:46:04 GMT
FYI I also just updated the wiki page with a Proposal D, aka Tucu plan,
which I think is essentially Proposal C but tabling JDK8 plans for now.

https://wiki.apache.org/hadoop/MovingToJdk7and8

Karthik, thanks for ringing in re: 2.5. I guess there's nothing urgently
required, the Jenkins stuff just needs to happen before 2.6. Still, I'm
happy to help with anything.

Thanks,
Andrew


On Fri, Jun 27, 2014 at 11:34 AM, Karthik Kambatla <kasha@cloudera.com>
wrote:

> As someone else already mentioned, we should announce one future release
> (may be, 2.5) as the last JDK6-based release before making the move to
> JDK7.
>
> I am comfortable calling 2.5 the last JDK6 release.
>
>
> On Fri, Jun 27, 2014 at 11:26 AM, Andrew Wang <andrew.wang@cloudera.com>
> wrote:
>
> > Hi all, responding to multiple messages here,
> >
> > Arun, thanks for the clarification regarding MR classpaths. It sounds
> like
> > the story there is improved and still improving.
> >
> > However, I think we still suffer from this at least on the HDFS side. We
> > have a single JAR for all of HDFS, and our clients need to have all the
> fun
> > deps like Guava on the classpath. I'm told Spark sticks a newer Guava at
> > the front of the classpath and the HDFS client still works okay, but this
> > is more happy coincidence than anything else. While we're leaking deps,
> > we're in a scary situation.
> >
> > API compat to me means that an app should be able to run on a new minor
> > version of Hadoop and not have anything break. MAPREDUCE-4421 sounds like
> > it allows you to run e.g. 2.3 MR jobs on a 2.4 YARN cluster, but what
> > should also be possible is running an HDFS 2.3 app with HDFS 2.4 JARs and
> > have nothing break. If we muck with the classpath, my understanding is
> that
> > this could break.
> >
> > Owen, bumping the minimum JDK version in a minor release like this should
> > be a one-time exception as Tucu stated. A number of people have pointed
> out
> > how painful a forced JDK upgrade is for end users, and it's not something
> > we should be springing on them in a minor release unless we're *very*
> > confident like in this case.
> >
> > Chris, thanks for bringing up the ecosystem. For CDH5, we standardized on
> > JDK7 across the CDH stack, so I think that's an indication that most
> > ecosystem projects are ready to make the jump. Is that sufficient in your
> > mind?
> >
> > For the record, I'm also +1 on the Tucu plan. Is it too late to do this
> for
> > 2.5? I'll offer to help out with some of the mechanics.
> >
> > Thanks,
> > Andrew
> >
> > On Wed, Jun 25, 2014 at 4:18 PM, Chris Nauroth <cnauroth@hortonworks.com
> >
> > wrote:
> >
> > > I understood the plan for avoiding JDK7-specific features in our code,
> > and
> > > your suggestion to add an extra Jenkins job is a great way to guard
> > against
> > > that.  The thing I haven't seen discussed yet is how downstream
> projects
> > > will continue to consume our built artifacts.  If a downstream project
> > > upgrades to pick up a bug fix, and the jar switches to 1.7 class files,
> > but
> > > their project is still building with 1.6, then it would be a nasty
> > > surprise.
> > >
> > > These are the options I see:
> > >
> > > 1. Make sure all other projects upgrade first.  This doesn't sound
> > > feasible, unless all other ecosystem projects have moved to JDK7
> already.
> > >  If not, then waiting on a single long pole project would hold up our
> > > migration indefinitely.
> > >
> > > 2. We switch to JDK7, but run javac with -target 1.6 until the whole
> > > ecosystem upgrades.  I find this undesirable, because in a certain
> sense,
> > > it still leaves a bit of 1.6 lingering in the project.  (I'll assume
> that
> > > end-of-life for JDK6 also means end-of-life for the 1.6 bytecode
> format.)
> > >
> > > 3. Just declare a clean break on some version (your earlier email said
> > 2.5)
> > > and start publishing artifacts built with JDK7 and no -target option.
> > >  Overall, this is my preferred option.  However, as a side effect, this
> > > sets us up for longer-term maintenance and patch releases off of the
> 2.4
> > > branch if a downstream project that's still on 1.6 needs to pick up a
> > > critical bug fix.
> > >
> > > Of course, this is all a moot point if all the downstream ecosystem
> > > projects have already made the switch to JDK7.  I don't know the status
> > of
> > > that off the top of my head.  Maybe someone else out there knows?  If
> > not,
> > > then I expect I can free up enough in a few weeks to volunteer for
> > tracking
> > > down that information.
> > >
> > > Chris Nauroth
> > > Hortonworks
> > > http://hortonworks.com/
> > >
> > >
> > >
> > > On Wed, Jun 25, 2014 at 3:12 PM, Alejandro Abdelnur <tucu@cloudera.com
> >
> > > wrote:
> > >
> > > > Chris,
> > > >
> > > > Compiling with jdk7 and doing javac -target 1.6 is not sufficient,
> you
> > > are
> > > > still using jdk7 libraries and you could use new APIs, thus breaking
> > jdk6
> > > > both at compile and runtime.
> > > >
> > > > you need to compile with jdk6 to ensure you are not running into that
> > > > scenario. that is why i was suggesting the nightly jdk6 build/test
> > > jenkins
> > > > job.
> > > >
> > > >
> > > > On Wed, Jun 25, 2014 at 2:04 PM, Chris Nauroth <
> > cnauroth@hortonworks.com
> > > >
> > > > wrote:
> > > >
> > > > > I'm also +1 for getting us to JDK7 within the 2.x line after
> reading
> > > the
> > > > > proposals and catching up on the discussion in this thread.
> > > > >
> > > > > Has anyone yet considered how to coordinate this change with
> > downstream
> > > > > projects?  Would we request downstream projects to upgrade to JDK7
> > > first
> > > > > before we make the move?  Would we switch to JDK7, but run javac
> > > -target
> > > > > 1.6 to maintain compatibility for downstream projects during an
> > interim
> > > > > period?
> > > > >
> > > > > Chris Nauroth
> > > > > Hortonworks
> > > > > http://hortonworks.com/
> > > > >
> > > > >
> > > > >
> > > > > On Wed, Jun 25, 2014 at 9:48 AM, Owen O'Malley <omalley@apache.org
> >
> > > > wrote:
> > > > >
> > > > > > On Tue, Jun 24, 2014 at 4:44 PM, Alejandro Abdelnur <
> > > tucu@cloudera.com
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > After reading this thread and thinking a bit about it,
I think
> it
> > > > > should
> > > > > > be
> > > > > > > OK such move up to JDK7 in Hadoop
> > > > > >
> > > > > >
> > > > > > I agree with Alejandro. Changing minimum JDKs is not an
> > incompatible
> > > > > change
> > > > > > and is fine in the 2 branch. (Although I think it is would *not*
> be
> > > > > > appropriate for a patch release.) Of course we need to do it
with
> > > > > > forethought and testing, but moving off of JDK 6, which is EOL'ed
> > is
> > > a
> > > > > good
> > > > > > thing. Moving to Java 8 as a minimum seems much too aggressive
> and
> > I
> > > > > would
> > > > > > push back on that.
> > > > > >
> > > > > > I'm also think that we need to let the dust settle on the Hadoop
> 2
> > > line
> > > > > for
> > > > > > a while before we talk about Hadoop 3. It seems that it has
only
> > been
> > > > in
> > > > > > the last 6 months that Hadoop 2 adoption has reached the main
> > stream
> > > > > users.
> > > > > > Our user community needs time to digest the changes in Hadoop
2.x
> > > > before
> > > > > we
> > > > > > fracture the community by starting to discuss Hadoop 3 releases.
> > > > > >
> > > > > > .. Owen
> > > > > >
> > > > >
> > > > > --
> > > > > CONFIDENTIALITY NOTICE
> > > > > NOTICE: This message is intended for the use of the individual or
> > > entity
> > > > to
> > > > > which it is addressed and may contain information that is
> > confidential,
> > > > > privileged and exempt from disclosure under applicable law. If the
> > > reader
> > > > > of this message is not the intended recipient, you are hereby
> > notified
> > > > that
> > > > > any printing, copying, dissemination, distribution, disclosure or
> > > > > forwarding of this communication is strictly prohibited. If you
> have
> > > > > received this communication in error, please contact the sender
> > > > immediately
> > > > > and delete it from your system. Thank You.
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Alejandro
> > > >
> > >
> > > --
> > > CONFIDENTIALITY NOTICE
> > > NOTICE: This message is intended for the use of the individual or
> entity
> > to
> > > which it is addressed and may contain information that is confidential,
> > > privileged and exempt from disclosure under applicable law. If the
> reader
> > > of this message is not the intended recipient, you are hereby notified
> > that
> > > any printing, copying, dissemination, distribution, disclosure or
> > > forwarding of this communication is strictly prohibited. If you have
> > > received this communication in error, please contact the sender
> > immediately
> > > and delete it from your system. Thank You.
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message