pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Julien Le Dem (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PIG-2478) Using Illustrate with Javascript UDF throws NEP
Date Wed, 20 Jun 2012 17:00:44 GMT

    [ https://issues.apache.org/jira/browse/PIG-2478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397629#comment-13397629
] 

Julien Le Dem commented on PIG-2478:
------------------------------------

The problem is that the input schema is not set in that case.
i.e.: Pig does not seem to call outputSchema in that case.
                
> Using Illustrate with Javascript UDF throws NEP
> -----------------------------------------------
>
>                 Key: PIG-2478
>                 URL: https://issues.apache.org/jira/browse/PIG-2478
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: 0.11
>         Environment: Mac OSX Lion 10.7.2, Hadoop (0.20.205.0), Pig (Apache Pig version
0.11.0-SNAPSHOT (r1230844).
>            Reporter: dan young
>
> When I try to run illustrate while using a Javascript UDF, I always get a NEP....but
if I do a dump/store, everything works fine.
> dans-MacBook-Pro:pig danoyoung$ pig -x local
> 2012-01-12 22:16:57,131 [main] INFO  org.apache.pig.Main - Logging error messages to:
/Users/danoyoung/projects/pig/pig_1326431817128.log
> 2012-01-12 22:16:57,386 [main] INFO  org.apache.pig.backend.hadoop.executionengine.HExecutionEngine
- Connecting to hadoop file system at: file:///
> 2012-01-12 22:16:57.483 java[27809:1903] Unable to load realm info from SCDynamicStore
> grunt> set io.sort.mb 500;
> grunt> register '/Users/danoyoung/projects/pig/udf/udf.js' using org.apache.pig.scripting.js.JsScriptEngine
as myfuncs;
> 2012-01-12 22:17:24,803 [main] INFO  org.apache.pig.scripting.js.JsScriptEngine - Register
scripting UDF: get_date_marker
> 2012-01-12 22:17:24,804 [main] INFO  org.apache.pig.scripting.js.JsScriptEngine - Register
scripting UDF: get_record
> grunt> register '/usr/local/pig/piggybank.jar';
> grunt> 
> grunt> a = LOAD '/Users/danoyoung/Downloads/adwords_KeywordDailyReport_1-1-2012_daily'
USING org.apache.pig.piggybank.storage.XMLLoader('row') AS (doc:chararray);
> grunt> 
> grunt> b = FOREACH a GENERATE FLATTEN(myfuncs.get_record(doc)) AS (dw_date_marker:int,ad_network_ad_group_key:long,ad_network_keyword_key:long,firstpagecpc:int,qualityscore:int,cost:float,position:float);
> grunt> describe b;
> b: {dw_date_marker: int,ad_network_ad_group_key: long,ad_network_keyword_key: long,firstpagecpc:
int,qualityscore: int,cost: float,position: float}
> grunt> 
> grunt> illustrate b;
> 2012-01-12 22:17:25,642 [main] INFO  org.apache.pig.backend.hadoop.executionengine.HExecutionEngine
- Connecting to hadoop file system at: file:///
> 2012-01-12 22:17:25,846 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler
- File concatenation threshold: 100 optimistic? false
> 2012-01-12 22:17:25,860 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
- MR plan size before optimization: 1
> 2012-01-12 22:17:25,860 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
- MR plan size after optimization: 1
> 2012-01-12 22:17:25,870 [main] INFO  org.apache.pig.tools.pigstats.ScriptState - Pig
script settings are added to the job
> 2012-01-12 22:17:25,887 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
> 2012-01-12 22:17:26,161 [main] INFO  org.apache.hadoop.mapreduce.lib.input.FileInputFormat
- Total input paths to process : 1
> 2012-01-12 22:17:26,721 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler
- File concatenation threshold: 100 optimistic? false
> 2012-01-12 22:17:26,723 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
- MR plan size before optimization: 1
> 2012-01-12 22:17:26,723 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
- MR plan size after optimization: 1
> 2012-01-12 22:17:26,724 [main] INFO  org.apache.pig.tools.pigstats.ScriptState - Pig
script settings are added to the job
> 2012-01-12 22:17:26,725 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
> java.lang.NullPointerException
> 	at org.apache.pig.scripting.js.JsFunction.exec(JsFunction.java:200)
> 	at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:225)
> 	at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:262)
> 	at org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.getNext(PhysicalOperator.java:334)
> 	at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.processPlan(POForEach.java:332)
> 	at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNext(POForEach.java:284)
> 	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.runPipeline(PigGenericMapBase.java:271)
> 	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:266)
> 	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:64)
> 	at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
> 	at org.apache.pig.pen.LocalMapReduceSimulator.launchPig(LocalMapReduceSimulator.java:194)
> 	at org.apache.pig.pen.ExampleGenerator.getData(ExampleGenerator.java:257)
> 	at org.apache.pig.pen.ExampleGenerator.getData(ExampleGenerator.java:238)
> 	at org.apache.pig.pen.LineageTrimmingVisitor.init(LineageTrimmingVisitor.java:103)
> 	at org.apache.pig.pen.LineageTrimmingVisitor.<init>(LineageTrimmingVisitor.java:98)
> 	at org.apache.pig.pen.ExampleGenerator.getExamples(ExampleGenerator.java:166)
> 	at org.apache.pig.PigServer.getExamples(PigServer.java:1202)
> 	at org.apache.pig.tools.grunt.GruntParser.processIllustrate(GruntParser.java:698)
> 	at org.apache.pig.tools.pigscript.parser.PigScriptParser.Illustrate(PigScriptParser.java:591)
> 	at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:306)
> 	at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:188)
> 	at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164)
> 	at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
> 	at org.apache.pig.Main.run(Main.java:523)
> 	at org.apache.pig.Main.main(Main.java:148)
> 2012-01-12 22:17:26,767 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2997: Encountered
IOException. Exception : null
> Details at logfile: /Users/danoyoung/projects/pig/pig_1326431817128.log
> grunt> 
> Here's the log details:
> Pig Stack Trace
> ---------------
> ERROR 2997: Encountered IOException. Exception : null
> java.io.IOException: Exception : null
>         at org.apache.pig.PigServer.getExamples(PigServer.java:1208)
>         at org.apache.pig.tools.grunt.GruntParser.processIllustrate(GruntParser.java:698)
>         at org.apache.pig.tools.pigscript.parser.PigScriptParser.Illustrate(PigScriptParser.java:591)
>         at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:306)
>         at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:188)
>         at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164)
>         at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
>         at org.apache.pig.Main.run(Main.java:523)
>         at org.apache.pig.Main.main(Main.java:148)
> ================================================================================
> Here's the Javascript UDF:
> get_record.outputSchema = "dw_date_marker:int,ad_network_ad_group_key:long,ad_network_keyword_key:long,firstpagecpc:int,qualityscore:int,cost:float,position:float";
> function get_record(doc){
>     var rec = new Object;
>     rec.dt = doc.substring(doc.search(/date=/ig)+6,doc.search(/\" campaignid=/ig));
>      //rec.campaign_id = doc.substring(doc.search(/campaignid=/ig)+12,doc.search(/\"
adgroupid=/ig));
>      rec.ad_network_ad_group_key = doc.substring(doc.search(/adgroupid=/ig)+11,doc.search(/\"
keywordid=/ig));
>      rec.ad_network_keyword_key = doc.substring(doc.search(/keywordid=/ig)+11,doc.search(/\"
keyword=/ig));
>      rec.firstpagecpc = doc.substring(doc.search(/firstpagecpc=/ig)+14,doc.search(/\"
qualityscore=/ig))/1000000;
>      rec.qualityscore = doc.substring(doc.search(/qualityscore=/ig)+14,doc.search(/\"
imps=/ig));
>      rec.cost = doc.substring(doc.search(/cost=/ig)+6,doc.search(/\" pos=/ig))/1000000;
>      rec.position = doc.substring(doc.search(/pos=/ig)+5,doc.search(/\"><\/row/ig));
>      rec.dw_date_marker = get_date_marker(rec.dt);
>               
>      return {dw_date_marker:rec.dw_date_marker,
>           ad_network_ad_group_key:rec.ad_network_ad_group_key,
>           ad_network_keyword_key:rec.ad_network_keyword_key,
>           firstpagecpc:rec.firstpagecpc,
>           qualityscore:rec.qualityscore,
>           cost:rec.cost,
>           position:rec.position};
> }
> function get_date_marker(dt){
>      var dateMarkers = {
>      _2012_01_01:1204,
>      _2012_01_02:1205,
>      _2012_01_03:1206,
>      _2012_01_04:1207,
>      _2012_01_05:1208,
>      _2012_01_06:1209,
>      _2012_01_07:1210,
>      _2012_01_08:1211,
>      _2012_01_09:1212,
>      _2012_01_10:1213,
>      _2012_01_11:1214,
>      _2012_01_12:1215,
>      _2012_01_13:1216,
>      _2012_01_14:1217,
>      _2012_01_15:1218,
>      _2012_01_16:1219,
>      _2012_01_17:1220,
>      _2012_01_18:1221,
>      _2012_01_19:1222,
>      _2012_01_20:1223,
>      _2012_01_21:1224,
>      _2012_01_22:1225,
>      _2012_01_23:1226,
>      _2012_01_24:1227,
>      _2012_01_25:1228,
>      _2012_01_26:1229,
>      _2012_01_27:1230,
>      _2012_01_28:1231,
>      _2012_01_29:1232,
>      _2012_01_30:1233,
>      _2012_01_31:1234,
>      _2012_02_01:1235,
>      _2012_02_02:1236,
>      _2012_02_03:1237,
>      _2012_02_04:1238,
>      _2012_02_05:1239,
>      _2012_02_06:1240,
>      _2012_02_07:1241,
>      _2012_02_08:1242,
>      _2012_02_09:1243,
>      _2012_02_10:1244,
>      _2012_02_11:1245,
>      _2012_02_12:1246,
>      _2012_02_13:1247,
>      _2012_02_14:1248,
>      _2012_02_15:1249,
>      _2012_02_16:1250,
>      _2012_02_17:1251,
>      _2012_02_18:1252,
>      _2012_02_19:1253,
>      _2012_02_20:1254,
>      _2012_02_21:1255,
>      _2012_02_22:1256,
>      _2012_02_23:1257,
>      _2012_02_24:1258,
>      _2012_02_25:1259,
>      _2012_02_26:1260,
>      _2012_02_27:1261,
>      _2012_02_28:1262,
>      _2012_02_29:1263,
>      _2012_03_01:1264,
>      _2012_03_02:1265,
>      _2012_03_03:1266,
>      _2012_03_04:1267,
>      _2012_03_05:1268,
>      _2012_03_06:1269,
>      _2012_03_07:1270,
>      _2012_03_08:1271,
>      _2012_03_09:1272,
>      _2012_03_10:1273,
>      _2012_03_11:1274,
>      _2012_03_12:1275,
>      _2012_03_13:1276,
>      _2012_03_14:1277,
>      _2012_03_15:1278,
>      _2012_03_16:1279,
>      _2012_03_17:1280,
>      _2012_03_18:1281,
>      _2012_03_19:1282,
>      _2012_03_20:1283,
>      _2012_03_21:1284,
>      _2012_03_22:1285,
>      _2012_03_23:1286,
>      _2012_03_24:1287,
>      _2012_03_25:1288,
>      _2012_03_26:1289,
>      _2012_03_27:1290,
>      _2012_03_28:1291,
>      _2012_03_29:1292,
>      _2012_03_30:1293,
>      _2012_03_31:1294,
>      _2012_04_01:1295,
>      _2012_04_02:1296,
>      _2012_04_03:1297,
>      _2012_04_04:1298,
>      _2012_04_05:1299,
>      _2012_04_06:1300,
>      _2012_04_07:1301,
>      _2012_04_08:1302,
>      _2012_04_09:1303,
>      _2012_04_10:1304,
>      _2012_04_11:1305,
>      _2012_04_12:1306,
>      _2012_04_13:1307,
>      _2012_04_14:1308,
>      _2012_04_15:1309,
>      _2012_04_16:1310,
>      _2012_04_17:1311,
>      _2012_04_18:1312,
>      _2012_04_19:1313,
>      _2012_04_20:1314,
>      _2012_04_21:1315,
>      _2012_04_22:1316,
>      _2012_04_23:1317,
>      _2012_04_24:1318,
>      _2012_04_25:1319,
>      _2012_04_26:1320,
>      _2012_04_27:1321,
>      _2012_04_28:1322,
>      _2012_04_29:1323,
>      _2012_04_30:1324,
>      _2012_05_01:1325,
>      _2012_05_02:1326,
>      _2012_05_03:1327,
>      _2012_05_04:1328,
>      _2012_05_05:1329,
>      _2012_05_06:1330,
>      _2012_05_07:1331,
>      _2012_05_08:1332,
>      _2012_05_09:1333,
>      _2012_05_10:1334,
>      _2012_05_11:1335,
>      _2012_05_12:1336,
>      _2012_05_13:1337,
>      _2012_05_14:1338,
>      _2012_05_15:1339,
>      _2012_05_16:1340,
>      _2012_05_17:1341,
>      _2012_05_18:1342,
>      _2012_05_19:1343,
>      _2012_05_20:1344,
>      _2012_05_21:1345,
>      _2012_05_22:1346,
>      _2012_05_23:1347,
>      _2012_05_24:1348,
>      _2012_05_25:1349,
>      _2012_05_26:1350,
>      _2012_05_27:1351,
>      _2012_05_28:1352,
>      _2012_05_29:1353,
>      _2012_05_30:1354,
>      _2012_05_31:1355,
>      _2012_06_01:1356,
>      _2012_06_02:1357,
>      _2012_06_03:1358,
>      _2012_06_04:1359,
>      _2012_06_05:1360,
>      _2012_06_06:1361,
>      _2012_06_07:1362,
>      _2012_06_08:1363,
>      _2012_06_09:1364,
>      _2012_06_10:1365,
>      _2012_06_11:1366,
>      _2012_06_12:1367,
>      _2012_06_13:1368,
>      _2012_06_14:1369,
>      _2012_06_15:1370,
>      _2012_06_16:1371,
>      _2012_06_17:1372,
>      _2012_06_18:1373,
>      _2012_06_19:1374,
>      _2012_06_20:1375,
>      _2012_06_21:1376,
>      _2012_06_22:1377,
>      _2012_06_23:1378,
>      _2012_06_24:1379,
>      _2012_06_25:1380,
>      _2012_06_26:1381,
>      _2012_06_27:1382,
>      _2012_06_28:1383,
>      _2012_06_29:1384,
>      _2012_06_30:1385,
>      _2012_07_01:1386,
>      _2012_07_02:1387,
>      _2012_07_03:1388,
>      _2012_07_04:1389,
>      _2012_07_05:1390,
>      _2012_07_06:1391,
>      _2012_07_07:1392,
>      _2012_07_08:1393,
>      _2012_07_09:1394,
>      _2012_07_10:1395,
>      _2012_07_11:1396,
>      _2012_07_12:1397,
>      _2012_07_13:1398,
>      _2012_07_14:1399,
>      _2012_07_15:1400,
>      _2012_07_16:1401,
>      _2012_07_17:1402,
>      _2012_07_18:1403,
>      _2012_07_19:1404,
>      _2012_07_20:1405,
>      _2012_07_21:1406,
>      _2012_07_22:1407,
>      _2012_07_23:1408,
>      _2012_07_24:1409,
>      _2012_07_25:1410,
>      _2012_07_26:1411,
>      _2012_07_27:1412,
>      _2012_07_28:1413,
>      _2012_07_29:1414,
>      _2012_07_30:1415,
>      _2012_07_31:1416,
>      _2012_08_01:1417,
>      _2012_08_02:1418,
>      _2012_08_03:1419,
>      _2012_08_04:1420,
>      _2012_08_05:1421,
>      _2012_08_06:1422,
>      _2012_08_07:1423,
>      _2012_08_08:1424,
>      _2012_08_09:1425,
>      _2012_08_10:1426,
>      _2012_08_11:1427,
>      _2012_08_12:1428,
>      _2012_08_13:1429,
>      _2012_08_14:1430,
>      _2012_08_15:1431,
>      _2012_08_16:1432,
>      _2012_08_17:1433,
>      _2012_08_18:1434,
>      _2012_08_19:1435,
>      _2012_08_20:1436,
>      _2012_08_21:1437,
>      _2012_08_22:1438,
>      _2012_08_23:1439,
>      _2012_08_24:1440,
>      _2012_08_25:1441,
>      _2012_08_26:1442,
>      _2012_08_27:1443,
>      _2012_08_28:1444,
>      _2012_08_29:1445,
>      _2012_08_30:1446,
>      _2012_08_31:1447,
>      _2012_09_01:1448,
>      _2012_09_02:1449,
>      _2012_09_03:1450,
>      _2012_09_04:1451,
>      _2012_09_05:1452,
>      _2012_09_06:1453,
>      _2012_09_07:1454,
>      _2012_09_08:1455,
>      _2012_09_09:1456,
>      _2012_09_10:1457,
>      _2012_09_11:1458,
>      _2012_09_12:1459,
>      _2012_09_13:1460,
>      _2012_09_14:1461,
>      _2012_09_15:1462,
>      _2012_09_16:1463,
>      _2012_09_17:1464,
>      _2012_09_18:1465,
>      _2012_09_19:1466,
>      _2012_09_20:1467,
>      _2012_09_21:1468,
>      _2012_09_22:1469,
>      _2012_09_23:1470,
>      _2012_09_24:1471,
>      _2012_09_25:1472,
>      _2012_09_26:1473,
>      _2012_09_27:1474,
>      _2012_09_28:1475,
>      _2012_09_29:1476,
>      _2012_09_30:1477,
>      _2012_10_01:1478,
>      _2012_10_02:1479,
>      _2012_10_03:1480,
>      _2012_10_04:1481,
>      _2012_10_05:1482,
>      _2012_10_06:1483,
>      _2012_10_07:1484,
>      _2012_10_08:1485,
>      _2012_10_09:1486,
>      _2012_10_10:1487,
>      _2012_10_11:1488,
>      _2012_10_12:1489,
>      _2012_10_13:1490,
>      _2012_10_14:1491,
>      _2012_10_15:1492,
>      _2012_10_16:1493,
>      _2012_10_17:1494,
>      _2012_10_18:1495,
>      _2012_10_19:1496,
>      _2012_10_20:1497,
>      _2012_10_21:1498,
>      _2012_10_22:1499,
>      _2012_10_23:1500,
>      _2012_10_24:1501,
>      _2012_10_25:1502,
>      _2012_10_26:1503,
>      _2012_10_27:1504,
>      _2012_10_28:1505,
>      _2012_10_29:1506,
>      _2012_10_30:1507,
>      _2012_10_31:1508,
>      _2012_11_01:1509,
>      _2012_11_02:1510,
>      _2012_11_03:1511,
>      _2012_11_04:1512,
>      _2012_11_05:1513,
>      _2012_11_06:1514,
>      _2012_11_07:1515,
>      _2012_11_08:1516,
>      _2012_11_09:1517,
>      _2012_11_10:1518,
>      _2012_11_11:1519,
>      _2012_11_12:1520,
>      _2012_11_13:1521,
>      _2012_11_14:1522,
>      _2012_11_15:1523,
>      _2012_11_16:1524,
>      _2012_11_17:1525,
>      _2012_11_18:1526,
>      _2012_11_19:1527,
>      _2012_11_20:1528,
>      _2012_11_21:1529,
>      _2012_11_22:1530,
>      _2012_11_23:1531,
>      _2012_11_24:1532,
>      _2012_11_25:1533,
>      _2012_11_26:1534,
>      _2012_11_27:1535,
>      _2012_11_28:1536,
>      _2012_11_29:1537,
>      _2012_11_30:1538,
>      _2012_12_01:1539,
>      _2012_12_02:1540,
>      _2012_12_03:1541,
>      _2012_12_04:1542,
>      _2012_12_05:1543,
>      _2012_12_06:1544,
>      _2012_12_07:1545,
>      _2012_12_08:1546,
>      _2012_12_09:1547,
>      _2012_12_10:1548,
>      _2012_12_11:1549,
>      _2012_12_12:1550,
>      _2012_12_13:1551,
>      _2012_12_14:1552,
>      _2012_12_15:1553,
>      _2012_12_16:1554,
>      _2012_12_17:1555,
>      _2012_12_18:1556,
>      _2012_12_19:1557,
>      _2012_12_20:1558,
>      _2012_12_21:1559,
>      _2012_12_22:1560,
>      _2012_12_23:1561,
>      _2012_12_24:1562,
>      _2012_12_25:1563,
>      _2012_12_26:1564,
>      _2012_12_27:1565,
>      _2012_12_28:1566,
>      _2012_12_29:1567,
>      _2012_12_30:1568,
>      _2012_12_31:1569};
>      var rec = new Object;
>     rec.dt = dt.replace(/-/g, "_");
>     rec.date_marker = dateMarkers["_" + rec.dt];
>     return rec.date_marker;
> }

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message