pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Adam Szita (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PIG-5387) Test failures on JRE 11
Date Thu, 16 May 2019 13:55:00 GMT

    [ https://issues.apache.org/jira/browse/PIG-5387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16841368#comment-16841368
] 

Adam Szita commented on PIG-5387:
---------------------------------

So as far as I understand:
 * in the testing part you are injecting an URLClassLoader if we're running java11. Quite
hacky but not worse than the current implementation which uses reflection to invoke addURL
on URLClassLoader;
 * in Operator class I think using readObject like this will be fine, but please add javadoc
or comments to the readObject method that includes some reference to the JVM bug. We don't
want ppl to remove it by mistake :)

Having this in mind [^PIG-5387_2.patch] looks good to me.

> Test failures on JRE 11
> -----------------------
>
>                 Key: PIG-5387
>                 URL: https://issues.apache.org/jira/browse/PIG-5387
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: 0.17.0
>            Reporter: Nandor Kollar
>            Assignee: Nandor Kollar
>            Priority: Major
>         Attachments: PIG-5387_1.patch, PIG-5387_2.patch
>
>
> I tried to compile Pig with JDK 8 and execute the test with Java 11, and faced with several
test failures. For example TestCommit#testCheckin2 failed with the following exception:
> {code}
> 2019-05-08 16:06:09,712 WARN  [Thread-108] mapred.LocalJobRunner (LocalJobRunner.java:run(590))
- job_local1000317333_0003
> java.lang.Exception: java.io.IOException: Deserialization error: null
> 	at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:492)
> 	at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:552)
> Caused by: java.io.IOException: Deserialization error: null
> 	at org.apache.pig.impl.util.ObjectSerializer.deserialize(ObjectSerializer.java:62)
> 	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.setup(PigGenericMapBase.java:183)
> 	at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
> 	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799)
> 	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
> 	at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271)
> 	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> 	at java.base/java.lang.Thread.run(Thread.java:834)
> Caused by: java.lang.NullPointerException
> 	at org.apache.pig.impl.plan.Operator.hashCode(Operator.java:106)
> 	at java.base/java.util.HashMap.hash(HashMap.java:339)
> 	at java.base/java.util.HashMap.readObject(HashMap.java:1461)
> 	at java.base/jdk.internal.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
> 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> 	at java.base/java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1160)
> {code}
> It deserialization of one of the map plan failed, it appears we ran into [JDK-8201131|https://bugs.openjdk.java.net/browse/JDK-8201131].
I seems that the workaround in the issue report works, adding a readObject method to org.apache.pig.impl.plan.Operator:
> {code}
>     private void readObject(ObjectInputStream in) throws ClassNotFoundException, IOException
{
>         in.defaultReadObject();
>     }
> {code}
> solves the problem, however I'm not sure that this is the optimal solution.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message