pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "jiraposter@reviews.apache.org (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PIG-1883) Pig's progress estimation should account for parallel job executions
Date Mon, 02 May 2011 21:43:03 GMT

    [ https://issues.apache.org/jira/browse/PIG-1883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13027863#comment-13027863
] 

jiraposter@reviews.apache.org commented on PIG-1883:
----------------------------------------------------


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/683/#review631
-----------------------------------------------------------


This doesn't lend itself well to automated testing.  Any thoughts on how to test how the new
progress indicator does versus the existing one?  Have you run any initial tests to measure
this?


trunk/src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/Launcher.java
<https://reviews.apache.org/r/683/#comment1275>

    I don't understand the logic here.  Why is it 0% done if ANY job is waiting, etc.?  Some
of the jobs may be done and some partially done and some not even started.



trunk/src/org/apache/pig/impl/plan/OperatorPlan.java
<https://reviews.apache.org/r/683/#comment1276>

    This code shouldn't be in OperatorPlan.  We want to keep that as clean as possible.  Instead
you should build a new Walker type that can do this calculation.



trunk/src/org/apache/pig/impl/plan/OperatorPlan.java
<https://reviews.apache.org/r/683/#comment1277>

    You have tabs here and some other spots.  Please make sure you use 4 spaces rather than
tabs.



trunk/src/org/apache/pig/tools/pigstats/PigProgressNotificationListener.java
<https://reviews.apache.org/r/683/#comment1278>

    Why is a separate method needed here?  When users turn on the new progress indicator I
assume they don't get the old one too.  Given that the interfaces are the same it seems one
method should suffice here.



trunk/src/org/apache/pig/tools/pigstats/ScriptState.java
<https://reviews.apache.org/r/683/#comment1279>

    It looks like this comment got attached to the run method.  Also, the method has only
one parameter, but two are listed in the comment.


- Alan


On 2011-05-02 20:41:04, Alan Gates wrote:
bq.  
bq.  -----------------------------------------------------------
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/683/
bq.  -----------------------------------------------------------
bq.  
bq.  (Updated 2011-05-02 20:41:04)
bq.  
bq.  
bq.  Review request for pig.
bq.  
bq.  
bq.  Summary
bq.  -------
bq.  
bq.  This is Laukik's patch for PIG-1883
bq.  
bq.  
bq.  This addresses bug PIG-1883.
bq.      https://issues.apache.org/jira/browse/PIG-1883
bq.  
bq.  
bq.  Diffs
bq.  -----
bq.  
bq.    trunk/src/org/apache/pig/Main.java 1097661 
bq.    trunk/src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/Launcher.java
1097661 
bq.    trunk/src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/MapReduceLauncher.java
1097661 
bq.    trunk/src/org/apache/pig/impl/plan/OperatorPlan.java 1097661 
bq.    trunk/src/org/apache/pig/scripting/SyncProgressNotificationAdaptor.java 1097661 
bq.    trunk/src/org/apache/pig/tools/pigstats/PigProgressNotificationListener.java 1097661

bq.    trunk/src/org/apache/pig/tools/pigstats/ScriptState.java 1097661 
bq.    trunk/test/org/apache/pig/test/TestOperatorPlan.java 1097661 
bq.  
bq.  Diff: https://reviews.apache.org/r/683/diff
bq.  
bq.  
bq.  Testing
bq.  -------
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Alan
bq.  
bq.



> Pig's progress estimation should account for parallel job executions
> --------------------------------------------------------------------
>
>                 Key: PIG-1883
>                 URL: https://issues.apache.org/jira/browse/PIG-1883
>             Project: Pig
>          Issue Type: Improvement
>            Reporter: Laukik Chitnis
>            Assignee: Laukik Chitnis
>         Attachments: PIG-1883-2.patch
>
>
> Currently, Pig's progress estimation is based on the percentage of jobs completed out
of the total number of MR jobs. However, since the MR operators are arranged in a DAG (and
hence more than 1 job might be submitted for execution in parallel), the progress estimation
can be improved by considering the number of jobs in the critical path, instead of just the
total number of jobs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message