pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rohini Palaniswamy (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PIG-3376) Jython UDF invocation failure when importing java class in Cluster mode
Date Wed, 10 Jul 2013 16:33:50 GMT

    [ https://issues.apache.org/jira/browse/PIG-3376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13704725#comment-13704725
] 

Rohini Palaniswamy commented on PIG-3376:
-----------------------------------------

Can you give the full stacktrace from the map job?
                
> Jython UDF invocation failure when importing java class in Cluster mode
> -----------------------------------------------------------------------
>
>                 Key: PIG-3376
>                 URL: https://issues.apache.org/jira/browse/PIG-3376
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: 0.11
>         Environment: local, cluster
>            Reporter: Sergey
>
> Hi, I'm testing my scripts in local mode then I run them in production using oozie.
> Locally everything works fine. My pig version is 0.11
> When I run the same script in cluster mode, I do get exception on line where jython udf
is invoked. Here is my UDF, see it imports java class. This class is IN runtime, I give 100%:
> {code}
> from ru.pig.geo import GSMCellCalculator
> @outputSchema("centerLon:double, centerLat:double")
> def calculateDropShapedCenter(lon, lat, maxDist, angleWidth, azimuth):
>     print lon, lat, maxDist, angleWidth, azimuth
>     spatialCoord = GSMCellCalculator.getDropShapeCenter(lon, lat, float(maxDist), float(angleWidth),
float(azimuth))
>     return spatialCoord.longitude, spatialCoord.latitude
> {code}
> Here is a part of script:
> {code}
> register '$geoSpatialUdfs' using jython as udf;
> /*
> some code goes here....
> */
> gsmCellProj = FOREACH gsmCellFixed GENERATE
>                                          branchId,
>                                          cellId, lac,
>                                          lon, lat,
>                                          (int)azimuth, (int)midDist, (int)maxDist,
>                                          cellType, (int)angWidth, gen, startAng,
>                                          angWidthFixed, startAngFixed,
>                                          FLATTEN(udf.calculateDropShapedCenter(lon, lat,
midDist, angWidth, azimuth));
> {code}
> Here is the log STDOUT:
> {code}
> -07-10 17:33:30,729 [JobControl] INFO  org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil
 - Total input paths to process : 1
> 2013-07-10 17:33:30,729 [JobControl] INFO  org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil
 - Total input paths to process : 1
> 2013-07-10 17:33:30,737 [JobControl] INFO  org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil
 - Total input paths (combined) to process : 1
> 2013-07-10 17:33:30,737 [JobControl] INFO  org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil
 - Total input paths (combined) to process : 1
> 2013-07-10 17:33:31,562 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
 - HadoopJobId: job_201307101220_0154
> 2013-07-10 17:33:31,562 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
 - HadoopJobId: job_201307101220_0154
> 2013-07-10 17:33:31,563 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
 - Processing aliases cellUniqueLacCid,gsmCell,gsmCellFiltered,gsmCellFilteredGrp,gsmCellFixed,gsmCellProj
> 2013-07-10 17:33:31,563 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
 - Processing aliases cellUniqueLacCid,gsmCell,gsmCellFiltered,gsmCellFilteredGrp,gsmCellFixed,gsmCellProj
> 2013-07-10 17:33:31,563 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
 - detailed locations: M: gsmCell[3,10],gsmCell[-1,-1],gsmCellFiltered[20,18],cellUniqueLacCid[33,19],gsmCellFilteredGrp[32,21]
C: cellUniqueLacCid[33,19],gsmCellFilteredGrp[32,21] R: cellUniqueLacCid[33,19],gsmCellFixed[38,15],gsmCellProj[60,14]
> 2013-07-10 17:33:31,563 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
 - detailed locations: M: gsmCell[3,10],gsmCell[-1,-1],gsmCellFiltered[20,18],cellUniqueLacCid[33,19],gsmCellFilteredGrp[32,21]
C: cellUniqueLacCid[33,19],gsmCellFilteredGrp[32,21] R: cellUniqueLacCid[33,19],gsmCellFixed[38,15],gsmCellProj[60,14]
> 2013-07-10 17:33:31,563 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
 - More information at: http://prod-node016.lol.ru:50030/jobdetails.jsp?jobid=job_201307101220_0154
> 2013-07-10 17:33:31,563 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
 - More information at: http://prod-node016.lol.ru:50030/jobdetails.jsp?jobid=job_201307101220_0154
> Heart beat
> 2013-07-10 17:33:51,196 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
 - 14% complete
> 2013-07-10 17:33:51,196 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
 - 14% complete
> 2013-07-10 17:33:54,210 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
 - 34% complete
> 2013-07-10 17:33:54,210 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
 - 34% complete
> 2013-07-10 17:33:57,229 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
 - 50% complete
> 2013-07-10 17:33:57,229 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
 - 50% complete
> 2013-07-10 17:34:13,307 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
 - 83% complete
> 2013-07-10 17:34:13,307 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
 - 83% complete
> Heart beat
> 2013-07-10 17:34:31,899 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
 - 83% complete
> 2013-07-10 17:34:31,899 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
 - 83% complete
> Heart beat
> 2013-07-10 17:34:49,480 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
 - 83% complete
> 2013-07-10 17:34:49,480 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
 - 83% complete
> 2013-07-10 17:35:08,576 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
 - 83% complete
> 2013-07-10 17:35:08,576 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
 - 83% complete
> Heart beat
> 2013-07-10 17:35:26,170 [main] WARN  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
 - Ooops! Some job has failed! Specify -stop_on_failure if you want Pig to stop immediately
on failure.
> 2013-07-10 17:35:26,170 [main] WARN  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
 - Ooops! Some job has failed! Specify -stop_on_failure if you want Pig to stop immediately
on failure.
> 2013-07-10 17:35:26,170 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
 - job job_201307101220_0154 has failed! Stop running all dependent jobs
> 2013-07-10 17:35:26,170 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
 - job job_201307101220_0154 has failed! 
> {code}
> Here is the log with error:
> {code}
> org.apache.pig.backend.executionengine.ExecException: ERROR 0: Error executing function
> 	at org.apache.pig.scripting.jython.JythonFunction.exec(JythonFunction.java:120)
> 	at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:337)
> 	at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:376)
> 	at org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.getNext(PhysicalOperator.java:354)
> 	at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.processPlan(POForEach.java:372)
> 	at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNext(POForEach.java:297)
> 	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.runPipeline(PigGenericMapReduce.java:465)
> 	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.processOnePackage
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message