pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matt Martin (Updated) (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (PIG-2481) Illustrate throws NullPointerException when custom load function uses counters
Date Wed, 18 Jan 2012 16:54:40 GMT

     [ https://issues.apache.org/jira/browse/PIG-2481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Matt Martin updated PIG-2481:
-----------------------------

    Attachment: pig_1326905442471.log
                illustrate_bug.txt
                illustrate_bug.pig

One can use these files and the following command to recreate the bug:

pig -x local -p INPUT=/Users/hadoop/illustrate_bug.txt /Users/hadoop/illustrate_bug.pig
                
> Illustrate throws NullPointerException when custom load function uses counters
> ------------------------------------------------------------------------------
>
>                 Key: PIG-2481
>                 URL: https://issues.apache.org/jira/browse/PIG-2481
>             Project: Pig
>          Issue Type: Bug
>          Components: impl
>    Affects Versions: 0.9.1
>         Environment: Apache Pig 0.9.1
> JSON Loader from ElephantBird 2.1.1 (i.e. all changes through the following commit: https://github.com/kevinweil/elephant-bird/commit/13dda2a1de31844fe07bcf4a58e94fffcb9aa374)
>            Reporter: Matt Martin
>            Priority: Minor
>         Attachments: illustrate_bug.pig, illustrate_bug.txt, pig_1326905442471.log
>
>
> Pig throws the following NullPointerException while trying to use "illustrate" in conjunction
with ElephantBird's JSON loader:
> java.lang.NullPointerException
> 	at org.apache.hadoop.mapreduce.TaskInputOutputContext.getCounter(TaskInputOutputContext.java:84)
> 	at org.apache.pig.tools.pigstats.PigStatusReporter.getCounter(PigStatusReporter.java:55)
> 	at com.twitter.elephantbird.pig.util.PigCounterHelper.incrCounter(PigCounterHelper.java:54)
> 	at com.twitter.elephantbird.pig.load.JsonLoader.incrCounter(JsonLoader.java:91)
> 	at com.twitter.elephantbird.pig.load.JsonLoader.getNext(JsonLoader.java:55)
> 	at org.apache.pig.impl.io.ReadToEndLoader.getNextHelper(ReadToEndLoader.java:210)
> 	at org.apache.pig.impl.io.ReadToEndLoader.getNext(ReadToEndLoader.java:190)
> 	at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLoad.getNext(POLoad.java:129)
> 	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.runPipeline(PigGenericMapBase.java:267)
> 	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:262)
> 	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:64)
> 	at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
> 	at org.apache.pig.pen.LocalMapReduceSimulator.launchPig(LocalMapReduceSimulator.java:194)
> 	at org.apache.pig.pen.ExampleGenerator.getData(ExampleGenerator.java:257)
> 	at org.apache.pig.pen.ExampleGenerator.readBaseData(ExampleGenerator.java:222)
> 	at org.apache.pig.pen.ExampleGenerator.getExamples(ExampleGenerator.java:154)
> 	at org.apache.pig.PigServer.getExamples(PigServer.java:1245)
> 	at org.apache.pig.tools.grunt.GruntParser.processIllustrate(GruntParser.java:698)
> 	at org.apache.pig.tools.pigscript.parser.PigScriptParser.Illustrate(PigScriptParser.java:591)
> 	at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:306)
> 	at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:188)
> 	at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164)
> 	at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
> 	at org.apache.pig.Main.run(Main.java:553)
> 	at org.apache.pig.Main.main(Main.java:108)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> 	at java.lang.reflect.Method.invoke(Method.java:597)
> 	at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
> I have verified that this issue does not occur if the ElephantBird JSON loader is not
used so the issue seems to be related to the interaction between Pig and a Custom Loader (in
this case, it seems to be related to the way in which Pig's "illustrate" functionality handles
custom counters).  I've also verified that this issue does not occur with Pig 0.8.1 regardless
of whether the ElephantBird JSON loader is used or not, so this seems to be a Pig-specific
issue (as opposed to ElephantBird-related).
> I'll attach sample input and script for reference. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message