pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Doug Daniels (Updated) (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (PIG-2313) NPE in ILLUSTRATE trying to get StatusReporter in STORE
Date Mon, 03 Oct 2011 19:36:34 GMT

     [ https://issues.apache.org/jira/browse/PIG-2313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Doug Daniels updated PIG-2313:
------------------------------

    Attachment: PIG-2313-first_shot.patch
                PIG-2313-example.tar.gz
    
> NPE in ILLUSTRATE trying to get StatusReporter in STORE
> -------------------------------------------------------
>
>                 Key: PIG-2313
>                 URL: https://issues.apache.org/jira/browse/PIG-2313
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: 0.9.0, 0.10
>            Reporter: Doug Daniels
>         Attachments: PIG-2313-example.tar.gz, PIG-2313-first_shot.patch
>
>
> I'm seeing an NPE trying to do an illustrate on a script.  So far the simplest version
of the script that exhibits the issue is:
> {code}
> raw = LOAD 'data.txt' USING PigStorage() AS (x:int, y:int);
> filtered = FILTER raw BY x < 5;
> grouped = GROUP filtered BY x;
> counted = FOREACH grouped
>          GENERATE group AS x,
>                   COUNT(filtered) AS the_count;
> rmf output;
> STORE counted INTO 'output';
> {code}
> I had to pass a few nested Exceptions along to get it, but the bottom stack trace looks
like:
> {code}
> Caused by: java.lang.NullPointerException
> 	at org.apache.hadoop.mapreduce.TaskInputOutputContext.getCounter(TaskInputOutputContext.java:88)
> 	at org.apache.pig.tools.pigstats.PigStatusReporter.getCounter(PigStatusReporter.java:60)
> 	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReducePOStoreImpl.createRecordCounter(MapReducePOStoreImpl.java:121)
> 	at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POStore.setUp(POStore.java:108)
> 	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.cleanup(PigGenericMapReduce.java:525)
> 	at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:178)
> 	at org.apache.pig.pen.LocalMapReduceSimulator.launchPig(LocalMapReduceSimulator.java:222)
> 	at org.apache.pig.pen.ExampleGenerator.getData(ExampleGenerator.java:257)
> 	at org.apache.pig.pen.ExampleGenerator.getData(ExampleGenerator.java:275)
> 	at org.apache.pig.pen.LineageTrimmingVisitor.checkNewBaseData(LineageTrimmingVisitor.java:418)
> 	... 21 more
> {code}
> It looks like the IllustratorContext in the hadoop20 PigMapReduce.java shim (line 73)
is getting setup with a null StatusReporter.  This seems to be for the Reduce phase.  On the
other hand, the PigMapBase.java sets up the IllustratorContext with an IllustratorDummyReporter
for the Map phase.
> Eventually when the code in MapReducePOStoreImpl line 121 tries to get the reporter,
it fails with the NPE.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message