pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Olga Natkovich (JIRA)" <j...@apache.org>
Subject [jira] Commented: (PIG-1829) "0" value seen in PigStat's map/reduce runtime, even when the job is successful
Date Fri, 28 Jan 2011 20:01:05 GMT

    [ https://issues.apache.org/jira/browse/PIG-1829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12988203#action_12988203
] 

Olga Natkovich commented on PIG-1829:
-------------------------------------

I agree that having standard API is useful. I don't think I like the idea of reading current
structure because this will make it difficult to run against concurrent versions of Hadoop.

I think we need to do further investigation and find other alternatives or just say that this
will resolve once we have a reasonable support form Hadoop

> "0" value seen in PigStat's map/reduce runtime, even when the job is successful
> -------------------------------------------------------------------------------
>
>                 Key: PIG-1829
>                 URL: https://issues.apache.org/jira/browse/PIG-1829
>             Project: Pig
>          Issue Type: Improvement
>    Affects Versions: 0.8.0
>            Reporter: Thejas M Nair
>             Fix For: 0.9.0
>
>
> Pig runtime calls JobClient.getMapTaskReports(jobId) and JobClient.getReduceTaskReports(jobId)
to get statistics about numbers of maps/reducers, as well as max/min/avg time of these tasks.
But from time to time, these calls return empty lists. When that happens pig is reports 0
values for the stats. 
> The jobtracker keeps the stats information only for a limited duration based on the configuration
parameters  mapred.jobtracker.completeuserjobs.maximum and mapred.job.tracker.retiredjobs.cache.size.
Since pig collects the stats after jobs have finished running, it is possible that the stats
for the initial jobs are no longer available. To have better chances of getting the stats,
it should be collected as soon as the job is over. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message