pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daniel Dai (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (PIG-3593) Import jython standard module fail on cluster
Date Thu, 28 Nov 2013 22:53:35 GMT

     [ https://issues.apache.org/jira/browse/PIG-3593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Daniel Dai resolved PIG-3593.
-----------------------------

      Resolution: Fixed
    Hadoop Flags: Reviewed

In my case, even register fail. job.jar always take precedence.

Patch committed to 0.12 branch and trunk. Thanks Rohini for quick review!

> Import jython standard module fail on cluster
> ---------------------------------------------
>
>                 Key: PIG-3593
>                 URL: https://issues.apache.org/jira/browse/PIG-3593
>             Project: Pig
>          Issue Type: Bug
>          Components: impl
>            Reporter: Daniel Dai
>            Assignee: Daniel Dai
>             Fix For: 0.12.1
>
>         Attachments: PIG-3593-1.patch
>
>
> The following script fail on cluster:
> {code}
> import urllib
> @outputSchema("url:chararray")
> def urlDecode(str):
>     return urllib.unquote_plus( str )
> {code}
> {code}
> register '126.py' using jython as myfuncs;
> a = load 'studenttab10k' using PigStorage() as (name:chararray, age:int, gpa:double);
> b = foreach a generate myfuncs.urlDecode(name);
> dump b;
> {code}
> Error stack:
> java.io.IOException: Deserialization error: could not instantiate 'org.apache.pig.scripting.jython.JythonFunction'
with arguments '[127.py, resplit]'
> 	at org.apache.pig.impl.util.ObjectSerializer.deserialize(ObjectSerializer.java:59)
> 	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.setup(PigGenericMapBase.java:180)
> 	at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)
> 	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
> 	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
> 	at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:394)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
> 	at org.apache.hadoop.mapred.Child.main(Child.java:249)
> Caused by: java.lang.RuntimeException: could not instantiate 'org.apache.pig.scripting.jython.JythonFunction'
with arguments '[127.py, resplit]'
> 	at org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:727)
> 	at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.instantiateFunc(POUserFunc.java:126)
> 	at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.readObject(POUserFunc.java:567)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> 	at java.lang.reflect.Method.invoke(Method.java:597)
> 	at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:979)
> 	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1873)
> 	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1777)
> 	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
> 	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:349)
> 	at java.util.ArrayList.readObject(ArrayList.java:593)
> 	at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> 	at java.lang.reflect.Method.invoke(Method.java:597)
> 	at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:979)
> 	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1873)
> 	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1777)
> 	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
> 	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:349)
> 	at java.util.HashMap.readObject(HashMap.java:1030)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> 	at java.lang.reflect.Method.invoke(Method.java:597)
> 	at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:979)
> 	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1873)
> 	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1777)
> 	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
> 	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1970)
> 	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1895)
> 	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1777)
> 	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
> 	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1970)
> 	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1895)
> 	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1777)
> 	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
> 	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:349)
> 	at java.util.ArrayList.readObject(ArrayList.java:593)
> 	at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> 	at java.lang.reflect.Method.invoke(Method.java:597)
> 	at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:979)
> 	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1873)
> 	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1777)
> 	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
> 	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1970)
> 	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1895)
> 	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1777)
> 	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
> 	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:349)
> 	at java.util.HashMap.readObject(HashMap.java:1030)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> 	at java.lang.reflect.Method.invoke(Method.java:597)
> 	at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:979)
> 	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1873)
> 	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1777)
> 	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
> 	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1970)
> 	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1895)
> 	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1777)
> 	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
> 	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:349)
> 	at org.apache.pig.impl.util.ObjectSerializer.deserialize(ObjectSerializer.java:57)
> 	... 9 more
> Caused by: java.lang.reflect.InvocationTargetException
> 	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> 	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> 	at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> 	at org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:695)
> 	... 75 more
> Caused by: java.lang.IllegalStateException: Could not initialize: 127.py
> 	at org.apache.pig.scripting.jython.JythonFunction.<init>(JythonFunction.java:92)
> 	... 80 more
> Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 1121: Python Error.
Traceback (most recent call last):
>   File "127.py", line 2, in <module>
>     import re
>   File "__pyclasspath__/re$py.class", line 279, in <module>
> java.lang.ArrayIndexOutOfBoundsException: 10
> 	at org.python.objectweb.asm.ClassReader.a(Unknown Source)
> 	at org.python.objectweb.asm.ClassReader.accept(Unknown Source)
> 	at org.python.objectweb.asm.ClassReader.accept(Unknown Source)
> 	at org.python.core.AnnotationReader.<init>(AnnotationReader.java:44)
> 	at org.python.core.imp.readCode(imp.java:219)
> 	at org.python.core.util.importer.getModuleCode(importer.java:202)
> 	at org.python.core.util.importer.importer_load_module(importer.java:95)
> 	at org.python.core.ClasspathPyImporter.ClasspathPyImporter_load_module(ClasspathPyImporter.java:63)
> 	at org.python.core.ClasspathPyImporter$ClasspathPyImporter_load_module_exposer.__call__(Unknown
Source)
> 	at org.python.core.PyBuiltinMethodNarrow.__call__(PyBuiltinMethodNarrow.java:47)
> 	at org.python.core.imp.loadFromLoader(imp.java:518)
> 	at org.python.core.imp.find_module(imp.java:472)
> 	at org.python.core.imp.import_next(imp.java:718)
> 	at org.python.core.imp.import_module_level(imp.java:827)
> 	at org.python.core.imp.importName(imp.java:917)
> 	at org.python.core.ImportFunction.__call__(__builtin__.java:1220)
> 	at org.python.core.PyObject.__call__(PyObject.java:357)
> 	at org.python.core.__builtin__.__import__(__builtin__.java:1173)
> 	at org.python.core.imp.importOne(imp.java:936)
> 	at re$py.f$0(/home/frank/hg/jython/jython/dist/Lib/re.py:289)
> 	at re$py.call_function(/home/frank/hg/jython/jython/dist/Lib/re.py)
> 	at org.python.core.PyTableCode.call(PyTableCode.java:165)
> 	at org.python.core.PyCode.call(PyCode.java:18)
> 	at org.python.core.imp.createFromCode(imp.java:391)
> 	at org.python.core.util.importer.importer_load_module(importer.java:109)
> 	at org.python.core.ClasspathPyImporter.ClasspathPyImporter_load_module(ClasspathPyImporter.java:63)
> 	at org.python.core.ClasspathPyImporter$ClasspathPyImporter_load_module_exposer.__call__(Unknown
Source)
> 	at org.python.core.PyBuiltinMethodNarrow.__call__(PyBuiltinMethodNarrow.java:47)
> 	at org.python.core.imp.loadFromLoader(imp.java:518)
> 	at org.python.core.imp.find_module(imp.java:472)
> 	at org.python.core.imp.import_next(imp.java:718)
> 	at org.python.core.imp.import_module_level(imp.java:827)
> 	at org.python.core.imp.importName(imp.java:917)
> 	at org.python.core.ImportFunction.__call__(__builtin__.java:1220)
> 	at org.python.core.PyObject.__call__(PyObject.java:357)
> 	at org.python.core.__builtin__.__import__(__builtin__.java:1173)
> 	at org.python.core.imp.importOne(imp.java:936)
> 	at org.python.pycode._pyx3.f$0(127.py:3)
> 	at org.python.pycode._pyx3.call_function(127.py)
> 	at org.python.core.PyTableCode.call(PyTableCode.java:165)
> 	at org.python.core.PyCode.call(PyCode.java:18)
> 	at org.python.core.Py.runCode(Py.java:1275)
> 	at org.python.util.PythonInterpreter.execfile(PythonInterpreter.java:235)
> 	at org.apache.pig.scripting.jython.JythonScriptEngine$Interpreter.execfile(JythonScriptEngine.java:217)
> 	at org.apache.pig.scripting.jython.JythonScriptEngine$Interpreter.init(JythonScriptEngine.java:163)
> 	at org.apache.pig.scripting.jython.JythonScriptEngine.getFunction(JythonScriptEngine.java:388)
> 	at org.apache.pig.scripting.jython.JythonFunction.<init>(JythonFunction.java:55)
> 	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> 	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> 	at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> 	at org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:695)
> 	at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.instantiateFunc(POUserFunc.java:126)
> 	at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.readObject(POUserFunc.java:567)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> 	at java.lang.reflect.Method.invoke(Method.java:597)
> 	at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:979)
> 	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1873)
> 	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1777)
> 	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
> 	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:349)
> 	at java.util.ArrayList.readObject(ArrayList.java:593)
> 	at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> 	at java.lang.reflect.Method.invoke(Method.java:597)
> 	at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:979)
> 	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1873)
> 	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1777)
> 	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
> 	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:349)
> 	at java.util.HashMap.readObject(HashMap.java:1030)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> 	at java.lang.reflect.Method.invoke(Method.java:597)
> 	at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:979)
> 	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1873)
> 	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1777)
> 	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
> 	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1970)
> 	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1895)
> 	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1777)
> 	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
> 	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1970)
> 	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1895)
> 	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1777)
> 	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
> 	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:349)
> 	at java.util.ArrayList.readObject(ArrayList.java:593)
> 	at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> 	at java.lang.reflect.Method.invoke(Method.java:597)
> 	at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:979)
> 	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1873)
> 	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1777)
> 	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
> 	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1970)
> 	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1895)
> 	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1777)
> 	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
> 	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:349)
> 	at java.util.HashMap.readObject(HashMap.java:1030)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> 	at java.lang.reflect.Method.invoke(Method.java:597)
> 	at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:979)
> 	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1873)
> 	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1777)
> 	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
> 	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1970)
> 	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1895)
> 	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1777)
> 	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
> 	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:349)
> 	at org.apache.pig.impl.util.ObjectSerializer.deserialize(ObjectSerializer.java:57)
> 	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.setup(PigGenericMapBase.java:180)
> 	at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)
> 	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
> 	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
> 	at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:394)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
> 	at org.apache.hadoop.mapred.Child.main(Child.java:249)
> Seems objectweb asm does not like repackaged jython libraries. 
> We have similar tests in TestScriptUDF with MiniCluster but those are running fine. Those
are added by PIG-1824 and that ticket seems solves the issue. But I check all released versions
since 0.10.0, none is working on cluster. I am not sure whether we fixed issue once in a while
then break it, or we didn't solve the issue on the cluster.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Mime
View raw message