hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daniel Dai (JIRA)" <j...@apache.org>
Subject [jira] Commented: (PIG-943) Pig crash when it cannot get counter from hadoop
Date Thu, 03 Sep 2009 23:02:57 GMT

    [ https://issues.apache.org/jira/browse/PIG-943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12751210#action_12751210
] 

Daniel Dai commented on PIG-943:
--------------------------------

Hi, Dmitriy,
All this patch does is checking for nulls. The extra work in the patch is when counters object
is null, we treat all counter as unknown, so we do not report confusing results to user. Which
part do you think "hacky"? Also parseLong is there for a long time and currently the only
problem we observe is RunningJob.getCounters is null. If this is your concern I can change
it to use stringToLong.

> Pig crash when it cannot get counter from hadoop
> ------------------------------------------------
>
>                 Key: PIG-943
>                 URL: https://issues.apache.org/jira/browse/PIG-943
>             Project: Pig
>          Issue Type: Bug
>          Components: impl
>    Affects Versions: 0.3.0
>            Reporter: Daniel Dai
>             Fix For: 0.4.0
>
>         Attachments: PIG-943-1.patch
>
>
> We see following call stacks in Pig:
> Case 1:
> Caused by: java.lang.NullPointerException
>         at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.computeWarningAggregate(MapReduceLauncher.java:390)
>         at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:238)
>         at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.execute(HExecutionEngine.java:265)
> Case 2:
> Caused by: java.lang.NullPointerException
>         at org.apache.pig.tools.pigstats.PigStats.accumulateMRStats(PigStats.java:150)
>         at org.apache.pig.tools.pigstats.PigStats.accumulateStats(PigStats.java:91)
>         at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:192)
>         at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.execute(HExecutionEngine.java:265)
> In both cases, hadoop jobs finishes without error. The cause of both problems is RunningJob.getCounters()
returns a null, and Pig do not currently check for that. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message