pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nandor Kollar (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PIG-5387) Test failures on JRE 11
Date Mon, 13 May 2019 08:15:00 GMT

    [ https://issues.apache.org/jira/browse/PIG-5387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16838353#comment-16838353

Nandor Kollar commented on PIG-5387:

[~knoguchi], [~rohini] the class loading related changes is an ugly hack which I'm not proud
of, but I didn't find any easy fix which is compatible with Java 11 as well as Java 8, and
this one seems to work. Fortunately it is only a test problem, not a production code issue.
Until Java 8 the system class loader ([AppClassLoader|https://github.com/openjdk/jdk/blob/jdk8-b120/jdk/src/share/classes/sun/misc/Launcher.java#L259])
extends URLClassLoader, thus casting and registering the jar at runtime works fine. This has
changes in Java 11 (I think since Java 9, but didn't verify this): system class loader no
longer extends URLClassLoader (see [here|https://github.com/openjdk/jdk11u/blob/master/src/java.base/share/classes/jdk/internal/loader/ClassLoaders.java#L151]).
I tried to create a new URLClassLoader and set the Thread's context class loader to it, but
Pig uses [getSystemResources|https://github.com/apache/pig/blob/trunk/src/org/apache/pig/PigServer.java#L575]
to locate the jars, which will ask system classloader, for the locations, and won't find resources
registered below it. That's why I came up with this hack: on Java 8 it will work as before,
on Java 11, it will register the new URLClassLoader as the parent of system class loader,
and will find the jars. I know that this is an ugly hack, I guess the correct answer to this
problem would be Java 11 module loading, but to remain compatible with Java 8 we should either
introduce several reflective invocations, or we should introduce something like Java 11 shim.
I don't think either option pays off for a single test fix like this one. The only incompatibility
in this case is the casting of system class loader to URLClassLoader, and since we don't do
it in any other place apart from these two test classes, I don't think it causes any issue
with registering commands in Pig. Refactoring registerNewResource to Utils class is a good
idea, as well as adding a comment which briefly describes the aforementioned situation.

> Test failures on JRE 11
> -----------------------
>                 Key: PIG-5387
>                 URL: https://issues.apache.org/jira/browse/PIG-5387
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: 0.17.0
>            Reporter: Nandor Kollar
>            Priority: Major
>         Attachments: PIG-5387_1.patch
> I tried to compile Pig with JDK 8 and execute the test with Java 11, and faced with several
test failures. For example TestCommit#testCheckin2 failed with the following exception:
> {code}
> 2019-05-08 16:06:09,712 WARN  [Thread-108] mapred.LocalJobRunner (LocalJobRunner.java:run(590))
- job_local1000317333_0003
> java.lang.Exception: java.io.IOException: Deserialization error: null
> 	at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:492)
> 	at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:552)
> Caused by: java.io.IOException: Deserialization error: null
> 	at org.apache.pig.impl.util.ObjectSerializer.deserialize(ObjectSerializer.java:62)
> 	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.setup(PigGenericMapBase.java:183)
> 	at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
> 	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799)
> 	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
> 	at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271)
> 	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> 	at java.base/java.lang.Thread.run(Thread.java:834)
> Caused by: java.lang.NullPointerException
> 	at org.apache.pig.impl.plan.Operator.hashCode(Operator.java:106)
> 	at java.base/java.util.HashMap.hash(HashMap.java:339)
> 	at java.base/java.util.HashMap.readObject(HashMap.java:1461)
> 	at java.base/jdk.internal.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
> 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> 	at java.base/java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1160)
> {code}
> It deserialization of one of the map plan failed, it appears we ran into [JDK-8201131|https://bugs.openjdk.java.net/browse/JDK-8201131].
I seems that the workaround in the issue report works, adding a readObject method to org.apache.pig.impl.plan.Operator:
> {code}
>     private void readObject(ObjectInputStream in) throws ClassNotFoundException, IOException
>         in.defaultReadObject();
>     }
> {code}
> solves the problem, however I'm not sure that this is the optimal solution.

This message was sent by Atlassian JIRA

View raw message