hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Richard Ding (JIRA)" <j...@apache.org>
Subject [jira] Commented: (PIG-908) Need a way to correlate MR jobs with Pig statements
Date Wed, 23 Jun 2010 19:04:55 GMT

    [ https://issues.apache.org/jira/browse/PIG-908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881828#action_12881828

Richard Ding commented on PIG-908:

It's hard to correlate MR jobs with line numbers in Pig script in the current implementation.
So we decided that the next best thing is to correlate MR jobs with aliases defined in Pig

PIG-1333 added "pig.alias" to the MR jobs so it can be viewed in Job xml. The value of "pig.alias"
is a comma-separated list of aliases since a MR job can be composed of several Pig statements.

> Need a way to correlate MR jobs with Pig statements
> ---------------------------------------------------
>                 Key: PIG-908
>                 URL: https://issues.apache.org/jira/browse/PIG-908
>             Project: Pig
>          Issue Type: Wish
>            Reporter: Dmitriy V. Ryaboy
>            Assignee: Richard Ding
>             Fix For: 0.8.0
> Complex Pig Scripts often generate many Map-Reduce jobs, especially with the recent introduction
of multi-store capabilities.
> For example, the first script in the Pig tutorial produces 5 MR jobs.
> There is currently very little support for debugging resulting jobs; if one of the MR
jobs fails, it is hard to figure out which part of the script it was responsible for. Explain
plans help, but even with the explain plan, a fair amount of effort (and sometimes, experimentation)
is required to correlate the failing MR job with the corresponding PigLatin statements.
> This ticket is created to discuss approaches to alleviating this problem.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message