hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sergio Pena <sergio.p...@cloudera.com>
Subject Re: [discuss] jdk8 support
Date Sat, 28 May 2016 02:41:21 GMT
I did the change to JDK8 on Ptest now.

Please let me tknow if there are issues with Java8.

- Sergio

On Fri, May 27, 2016 at 2:45 PM, Sergio Pena <sergio.pena@cloudera.com>
wrote:

> Thanks Mohit.
>
> I will plan to do the JDK8 change on Jenkins today EOD and monitor any
> issues through the weekend.
>
> - Sergio
>
>
> On Fri, May 27, 2016 at 2:58 AM, Mohit Sabharwal <mohit@cloudera.com>
> wrote:
>
>> Update on moving Hive2 tests to JDK8: I've addressed almost all test
>> failures in HIVE-13547 on java8 branch. There is one remaining
>> open item (HIVE-13834) which is currently assigned. Given current
>> state of flaky test runs, there might be few more.
>>
>> I will work with Sergio to merge the test fixes to master and switch
>> the Hive2 pre-commit job to use JDK8, hopefully sometime tomorrow.
>>
>> After Hive2 tests switch, if your patch sees ordering related test
>> failures in pre-commit run, it's likely JDK version related and you'll
>> need to build & re-run the test using JDK8. Number of such tests
>> should be relatively small.
>>
>> On Tue, Apr 19, 2016 at 10:43 AM, Mohit Sabharwal <mohit@cloudera.com>
>> wrote:
>>
>> > Created HIVE-13547 to track switching 2x tests to JDK8.
>> >
>> > On Wed, Apr 13, 2016 at 10:02 AM, Sergio Pena <sergio.pena@cloudera.com
>> >
>> > wrote:
>> >
>> >> I agree with such change as JDK7 is not longer supported.
>> >>
>> >> Changes on Jenkins and Hive PTest shouldn't be hard. We just need to
>> >> replace the path from java7 to java8. But I think we should fix all
>> JDK8
>> >> issues or most of them before doing the change or we will end up
>> having a
>> >> lot of failures on all JIRAs running pre-commit tests.
>> >>
>> >> +1 with the change.
>> >>
>> >> On Mon, Apr 11, 2016 at 9:34 PM, Siddharth Seth <sseth@apache.org>
>> wrote:
>> >>
>> >> > Option 3 sounds good. I'd ideally like to make JDK8 the minimum
>> >> requirement
>> >> > soon as well.
>> >> >
>> >> > On Mon, Apr 11, 2016 at 4:59 PM, Szehon Ho <szehon@cloudera.com>
>> wrote:
>> >> >
>> >> > > Sounds like a good plan, +1
>> >> > >
>> >> > > On Mon, Apr 11, 2016 at 4:31 PM, Mohit Sabharwal <
>> mohit@cloudera.com>
>> >> > > wrote:
>> >> > >
>> >> > > > Thanks, Ashutosh. Makes sense to keep the source and target
as
>> 1.7
>> >> > since
>> >> > > > we're not using any JDK8 specific features yet. So, option
(3)
>> >> > > essentially
>> >> > > > just means using JDK8 exclusively to build & test Hive2.
>> >> > > >
>> >> > > > On Sat, Apr 9, 2016 at 12:23 PM, Ashutosh Chauhan <
>> >> > hashutosh@apache.org>
>> >> > > > wrote:
>> >> > > >
>> >> > > > > Hi Mohit,
>> >> > > > >
>> >> > > > > I also think option 3 makes sense. We should strive
to keep
>> test
>> >> > matrix
>> >> > > > > small so that we can do fast QA runs for dev patches.
>> >> > > > > We can just use jdk7 to build & test Hive1 and jdk8
to build &
>> >> test
>> >> > > > Hive2.
>> >> > > > > However, I am not sure of explicitly dropping support
>> altogether
>> >> for
>> >> > > jdk7
>> >> > > > > on Hive2. We should make sure that in pom.xml java source
&
>> target
>> >> > > > > compatibility is still 1.7 (which already is the case
>> currently)
>> >> so
>> >> > > that
>> >> > > > > Hive2 is still compatible with jdk7. Unit tests as I
said we
>> can
>> >> run
>> >> > on
>> >> > > > > jdk8.
>> >> > > > >
>> >> > > > > Thanks,
>> >> > > > > Ashutosh
>> >> > > > >
>> >> > > > > On Fri, Apr 8, 2016 at 4:34 PM, Mohit Sabharwal <
>> >> mohit@cloudera.com>
>> >> > > > > wrote:
>> >> > > > >
>> >> > > > > > Hi folks,
>> >> > > > > >
>> >> > > > > > Oracle EOL'ed (ended public updates) for Java 7
in April
>> 2015.
>> >> > > > > > In order to support Java 8, we fixed quite a bunch
of tests
>> in
>> >> > > > > > HIVE-8607 (*) early last year. However, since our
pre-commit
>> >> tests
>> >> > > run
>> >> > > > > > on JDK7 only, any JDK8 test failures are getting
ignored. As
>> a
>> >> > > result,
>> >> > > > > > the count has crept up
>> >> > > > > > <
>> >> > > > > >
>> >> > > > >
>> >> > > >
>> >> > >
>> >> >
>> >>
>> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/HIVE-TRUNK-JAVA8/
>> >> > > > > > >
>> >> > > > > > back
>> >> > > > > > from zero to 125.
>> >> > > > > >
>> >> > > > > > Some options to address this:
>> >> > > > > >
>> >> > > > > > (1) Run pre-commit tests on both JDK7 and JDK8,
in both 1.x
>> >> > > > > > and 2.x: This will further slow down the pre-commit
run.
>> >> > > > > >
>> >> > > > > > (2) Alternate pre-commit test runs between JDK7
and JDK8 (in
>> >> > > > > > both 1.x and 2.x): It's a cheap hack. More messy,
though
>> >> failures
>> >> > > > > > won't get ignored.
>> >> > > > > >
>> >> > > > > > (3) Drop support for JDK7 in Hive 2.x, i.e. build
and test
>> Hive
>> >> 2.x
>> >> > > on
>> >> > > > > > JDK8 only. For Hive 1.x, continue running JDK7
pre-commit
>> tests.
>> >> > > > > > The pros are:
>> >> > > > > > - Simple test matrix.
>> >> > > > > > - No need to slow down test run or maintain version
specific
>> >> golden
>> >> > > > files
>> >> > > > > > (**).
>> >> > > > > > - 2.x looks like the logical place to move to JDK8.
>> >> > > > > > - Users transitioning to JDK8 for all other services
do not
>> >> have to
>> >> > > > > > maintain
>> >> > > > > > multiple java versions on the cluster.
>> >> > > > > >
>> >> > > > > > Option (3) looks most attractive to me.
>> >> > > > > >
>> >> > > > > > Moving to JDK8 also lines us up better for Java
9, which is
>> on
>> >> the
>> >> > > > > > horizon (Oracle will end public updates for Java
8 in Sep
>> 2017)
>> >> > > > > >
>> >> > > > > > Around 100 of the latest crop of failures are due
to one
>> cause
>> >> > > > > > (HIVE-13409).
>> >> > > > > > I can take a pass at triaging the rest if there
is consensus
>> >> around
>> >> > > > what
>> >> > > > > > to do overall.
>> >> > > > > >
>> >> > > > > > Thanks,
>> >> > > > > > Mohit
>> >> > > > > >
>> >> > > > > > (*) Most test failures are due to JDK8 using a
different hash
>> >> > > function
>> >> > > > > for
>> >> > > > > > HashMap  compared to JDK7. This results in (mostly
benign,
>> but
>> >> hard
>> >> > > > > > to debug) ordering differences in q-file output
related to
>> >> > > > serialization
>> >> > > > > > order of map entries, numbering of stages in query
plan, etc.
>> >> > > > > >
>> >> > > > > > (**) In some cases, hash function related ordering
>> differences
>> >> > > > originate
>> >> > > > > > in external libraries like Avro, antlr, json ,Thrift's
map,
>> etc.
>> >> > for
>> >> > > > > which
>> >> > > > > > code
>> >> > > > > > changes are either more involved or led to more
test
>> failures.
>> >> To
>> >> > > > > address
>> >> > > > > > this,
>> >> > > > > > we added support for version specific golden files
>> (HIVE-9109).
>> >> > > > > Currently,
>> >> > > > > > there
>> >> > > > > > are ~40 golden files with ".java1.8.out" extension.
>> >> > > > > >
>> >> > > > >
>> >> > > >
>> >> > >
>> >> >
>> >>
>> >
>> >
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message