hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Olga Natkovich (JIRA)" <j...@apache.org>
Subject [jira] Updated: (PIG-908) Need a way to correlate MR jobs with Pig statements
Date Mon, 23 Aug 2010 20:02:17 GMT

     [ https://issues.apache.org/jira/browse/PIG-908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Olga Natkovich updated PIG-908:
-------------------------------


With Pig 0.8.0 we print a summary of the execution that contains (among other things) how
aliases mapped to jobs. Example:

JobId   Maps    Reduces MaxMapTime      MinMapTIme      AvgMapTime      MaxReduceTime   MinReduceTime
  AvgReduceTime   Alias   Feature Outputs
job_201004271216_12712  1       1       3       3       3       12      12      12      B,C
    GROUP_BY,COMBINER
job_201004271216_12713  1       1       3       3       3       12      12      12      D
      SAMPLER
job_201004271216_12714  1       1       3       3       3       12      12      12      D
      ORDER_BY,COMBINER       hdfs://wilbur20.labs.corp.sp1.yahoo.com:9020/tmp/temp743703298/tmp-2019944040,


> Need a way to correlate MR jobs with Pig statements
> ---------------------------------------------------
>
>                 Key: PIG-908
>                 URL: https://issues.apache.org/jira/browse/PIG-908
>             Project: Pig
>          Issue Type: Wish
>            Reporter: Dmitriy V. Ryaboy
>            Assignee: Richard Ding
>             Fix For: 0.8.0
>
>
> Complex Pig Scripts often generate many Map-Reduce jobs, especially with the recent introduction
of multi-store capabilities.
> For example, the first script in the Pig tutorial produces 5 MR jobs.
> There is currently very little support for debugging resulting jobs; if one of the MR
jobs fails, it is hard to figure out which part of the script it was responsible for. Explain
plans help, but even with the explain plan, a fair amount of effort (and sometimes, experimentation)
is required to correlate the failing MR job with the corresponding PigLatin statements.
> This ticket is created to discuss approaches to alleviating this problem.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message