hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sergio Pena <sergio.p...@cloudera.com>
Subject Re: [discuss] jdk8 support
Date Fri, 03 Jun 2016 15:10:06 GMT
Hey Ashutosh,

I switched to JDK8 in master last weekend. Jenkins has been running all
tests in Java8 successfully for a week now.
There are still a few tests we need to fix, but so far is looking good.

Sergio

On Fri, Jun 3, 2016 at 12:02 AM, Ashutosh Chauhan <hashutosh@apache.org>
wrote:

> Now that branch-2.1 has been cut, I think its an opportune time to drop
> support of jdk7 altogether in master. That is we compile using jdk8 as
> well. What do others think?
>
> Ashutosh
>
> On Fri, May 27, 2016 at 7:41 PM, Sergio Pena <sergio.pena@cloudera.com>
> wrote:
>
> > I did the change to JDK8 on Ptest now.
> >
> > Please let me tknow if there are issues with Java8.
> >
> > - Sergio
> >
> > On Fri, May 27, 2016 at 2:45 PM, Sergio Pena <sergio.pena@cloudera.com>
> > wrote:
> >
> > > Thanks Mohit.
> > >
> > > I will plan to do the JDK8 change on Jenkins today EOD and monitor any
> > > issues through the weekend.
> > >
> > > - Sergio
> > >
> > >
> > > On Fri, May 27, 2016 at 2:58 AM, Mohit Sabharwal <mohit@cloudera.com>
> > > wrote:
> > >
> > >> Update on moving Hive2 tests to JDK8: I've addressed almost all test
> > >> failures in HIVE-13547 on java8 branch. There is one remaining
> > >> open item (HIVE-13834) which is currently assigned. Given current
> > >> state of flaky test runs, there might be few more.
> > >>
> > >> I will work with Sergio to merge the test fixes to master and switch
> > >> the Hive2 pre-commit job to use JDK8, hopefully sometime tomorrow.
> > >>
> > >> After Hive2 tests switch, if your patch sees ordering related test
> > >> failures in pre-commit run, it's likely JDK version related and you'll
> > >> need to build & re-run the test using JDK8. Number of such tests
> > >> should be relatively small.
> > >>
> > >> On Tue, Apr 19, 2016 at 10:43 AM, Mohit Sabharwal <mohit@cloudera.com
> >
> > >> wrote:
> > >>
> > >> > Created HIVE-13547 to track switching 2x tests to JDK8.
> > >> >
> > >> > On Wed, Apr 13, 2016 at 10:02 AM, Sergio Pena <
> > sergio.pena@cloudera.com
> > >> >
> > >> > wrote:
> > >> >
> > >> >> I agree with such change as JDK7 is not longer supported.
> > >> >>
> > >> >> Changes on Jenkins and Hive PTest shouldn't be hard. We just need
> to
> > >> >> replace the path from java7 to java8. But I think we should fix
all
> > >> JDK8
> > >> >> issues or most of them before doing the change or we will end
up
> > >> having a
> > >> >> lot of failures on all JIRAs running pre-commit tests.
> > >> >>
> > >> >> +1 with the change.
> > >> >>
> > >> >> On Mon, Apr 11, 2016 at 9:34 PM, Siddharth Seth <sseth@apache.org>
> > >> wrote:
> > >> >>
> > >> >> > Option 3 sounds good. I'd ideally like to make JDK8 the minimum
> > >> >> requirement
> > >> >> > soon as well.
> > >> >> >
> > >> >> > On Mon, Apr 11, 2016 at 4:59 PM, Szehon Ho <szehon@cloudera.com>
> > >> wrote:
> > >> >> >
> > >> >> > > Sounds like a good plan, +1
> > >> >> > >
> > >> >> > > On Mon, Apr 11, 2016 at 4:31 PM, Mohit Sabharwal <
> > >> mohit@cloudera.com>
> > >> >> > > wrote:
> > >> >> > >
> > >> >> > > > Thanks, Ashutosh. Makes sense to keep the source
and target
> as
> > >> 1.7
> > >> >> > since
> > >> >> > > > we're not using any JDK8 specific features yet.
So, option
> (3)
> > >> >> > > essentially
> > >> >> > > > just means using JDK8 exclusively to build &
test Hive2.
> > >> >> > > >
> > >> >> > > > On Sat, Apr 9, 2016 at 12:23 PM, Ashutosh Chauhan
<
> > >> >> > hashutosh@apache.org>
> > >> >> > > > wrote:
> > >> >> > > >
> > >> >> > > > > Hi Mohit,
> > >> >> > > > >
> > >> >> > > > > I also think option 3 makes sense. We should
strive to keep
> > >> test
> > >> >> > matrix
> > >> >> > > > > small so that we can do fast QA runs for dev
patches.
> > >> >> > > > > We can just use jdk7 to build & test Hive1
and jdk8 to
> build
> > &
> > >> >> test
> > >> >> > > > Hive2.
> > >> >> > > > > However, I am not sure of explicitly dropping
support
> > >> altogether
> > >> >> for
> > >> >> > > jdk7
> > >> >> > > > > on Hive2. We should make sure that in pom.xml
java source &
> > >> target
> > >> >> > > > > compatibility is still 1.7 (which already
is the case
> > >> currently)
> > >> >> so
> > >> >> > > that
> > >> >> > > > > Hive2 is still compatible with jdk7. Unit
tests as I said
> we
> > >> can
> > >> >> run
> > >> >> > on
> > >> >> > > > > jdk8.
> > >> >> > > > >
> > >> >> > > > > Thanks,
> > >> >> > > > > Ashutosh
> > >> >> > > > >
> > >> >> > > > > On Fri, Apr 8, 2016 at 4:34 PM, Mohit Sabharwal
<
> > >> >> mohit@cloudera.com>
> > >> >> > > > > wrote:
> > >> >> > > > >
> > >> >> > > > > > Hi folks,
> > >> >> > > > > >
> > >> >> > > > > > Oracle EOL'ed (ended public updates)
for Java 7 in April
> > >> 2015.
> > >> >> > > > > > In order to support Java 8, we fixed
quite a bunch of
> tests
> > >> in
> > >> >> > > > > > HIVE-8607 (*) early last year. However,
since our
> > pre-commit
> > >> >> tests
> > >> >> > > run
> > >> >> > > > > > on JDK7 only, any JDK8 test failures
are getting ignored.
> > As
> > >> a
> > >> >> > > result,
> > >> >> > > > > > the count has crept up
> > >> >> > > > > > <
> > >> >> > > > > >
> > >> >> > > > >
> > >> >> > > >
> > >> >> > >
> > >> >> >
> > >> >>
> > >>
> >
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/HIVE-TRUNK-JAVA8/
> > >> >> > > > > > >
> > >> >> > > > > > back
> > >> >> > > > > > from zero to 125.
> > >> >> > > > > >
> > >> >> > > > > > Some options to address this:
> > >> >> > > > > >
> > >> >> > > > > > (1) Run pre-commit tests on both JDK7
and JDK8, in both
> 1.x
> > >> >> > > > > > and 2.x: This will further slow down
the pre-commit run.
> > >> >> > > > > >
> > >> >> > > > > > (2) Alternate pre-commit test runs between
JDK7 and JDK8
> > (in
> > >> >> > > > > > both 1.x and 2.x): It's a cheap hack.
More messy, though
> > >> >> failures
> > >> >> > > > > > won't get ignored.
> > >> >> > > > > >
> > >> >> > > > > > (3) Drop support for JDK7 in Hive 2.x,
i.e. build and
> test
> > >> Hive
> > >> >> 2.x
> > >> >> > > on
> > >> >> > > > > > JDK8 only. For Hive 1.x, continue running
JDK7 pre-commit
> > >> tests.
> > >> >> > > > > > The pros are:
> > >> >> > > > > > - Simple test matrix.
> > >> >> > > > > > - No need to slow down test run or maintain
version
> > specific
> > >> >> golden
> > >> >> > > > files
> > >> >> > > > > > (**).
> > >> >> > > > > > - 2.x looks like the logical place to
move to JDK8.
> > >> >> > > > > > - Users transitioning to JDK8 for all
other services do
> not
> > >> >> have to
> > >> >> > > > > > maintain
> > >> >> > > > > > multiple java versions on the cluster.
> > >> >> > > > > >
> > >> >> > > > > > Option (3) looks most attractive to me.
> > >> >> > > > > >
> > >> >> > > > > > Moving to JDK8 also lines us up better
for Java 9, which
> is
> > >> on
> > >> >> the
> > >> >> > > > > > horizon (Oracle will end public updates
for Java 8 in Sep
> > >> 2017)
> > >> >> > > > > >
> > >> >> > > > > > Around 100 of the latest crop of failures
are due to one
> > >> cause
> > >> >> > > > > > (HIVE-13409).
> > >> >> > > > > > I can take a pass at triaging the rest
if there is
> > consensus
> > >> >> around
> > >> >> > > > what
> > >> >> > > > > > to do overall.
> > >> >> > > > > >
> > >> >> > > > > > Thanks,
> > >> >> > > > > > Mohit
> > >> >> > > > > >
> > >> >> > > > > > (*) Most test failures are due to JDK8
using a different
> > hash
> > >> >> > > function
> > >> >> > > > > for
> > >> >> > > > > > HashMap  compared to JDK7. This results
in (mostly
> benign,
> > >> but
> > >> >> hard
> > >> >> > > > > > to debug) ordering differences in q-file
output related
> to
> > >> >> > > > serialization
> > >> >> > > > > > order of map entries, numbering of stages
in query plan,
> > etc.
> > >> >> > > > > >
> > >> >> > > > > > (**) In some cases, hash function related
ordering
> > >> differences
> > >> >> > > > originate
> > >> >> > > > > > in external libraries like Avro, antlr,
json ,Thrift's
> map,
> > >> etc.
> > >> >> > for
> > >> >> > > > > which
> > >> >> > > > > > code
> > >> >> > > > > > changes are either more involved or led
to more test
> > >> failures.
> > >> >> To
> > >> >> > > > > address
> > >> >> > > > > > this,
> > >> >> > > > > > we added support for version specific
golden files
> > >> (HIVE-9109).
> > >> >> > > > > Currently,
> > >> >> > > > > > there
> > >> >> > > > > > are ~40 golden files with ".java1.8.out"
extension.
> > >> >> > > > > >
> > >> >> > > > >
> > >> >> > > >
> > >> >> > >
> > >> >> >
> > >> >>
> > >> >
> > >> >
> > >>
> > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message