Mailing-List: contact pig-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: pig-dev@incubator.apache.org
Message-ID: <87883203.1213791645561.JavaMail.jira@brutus>
Date: Wed, 18 Jun 2008 05:20:45 -0700 (PDT)
From: "Pi Song (JIRA)" <jira@apache.org>
To: pig-dev@incubator.apache.org
Subject: [jira] Commented: (PIG-273) Need to optimize the ways splits are
 handled, both in the top level plan and in nested plans.
In-Reply-To: <1706380616.1213717784958.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/PIG-273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12605923#action_12605923 ] 

Pi Song commented on PIG-273:
-----------------------------

I assume this is the next step after pipeline rework.

We can keep inner plans as DAGs so that we still maintain all the knowledge needed for optimization (One might say we can store inner plans separately and merge them but I think that's more difficult).  

Idea on how to optimize will come soon!

> Need to optimize the ways splits are handled, both in the top level plan and in nested plans.
> ---------------------------------------------------------------------------------------------
>
>                 Key: PIG-273
>                 URL: https://issues.apache.org/jira/browse/PIG-273
>             Project: Pig
>          Issue Type: Improvement
>          Components: impl
>            Reporter: Alan Gates
>            Priority: Minor
>
> Currently, in the new pipeline rework (see PIG-157), splits in the data flow are not handled efficiently.  
> In the top level plans splits cause all the output data to be written to hdfs and then reread by each leg of the split.  This forces both a read/write and a new map/reduce pass when it is not always necessary.  For example, consider:
> A = load 'myfile';
> split A into B  if $0 < 100, C if $0 >= 100;
> B1 = group B by $0;
> ...
> C1 = group B by $1;
> ...
> In this case A will be loaded, and then immediately stored again.  Then a plan will be executed that handles the B* part of the script, and then another executed that will handle the C* part of the script.
> In nested plans, each projection of the generate is computed separately, even if they share common steps in the plan.  For example:
> B = group A by $0;
> C= foreach B {
>     C1 = distinct $1;
>     C2 = filter C1 by $1 > 0;
>     generate group, COUNT(C1), COUNT(C2);
> }
> That will currently be executed with two nested plans, distinct->COUNT(C1) and distinct->filter->COUNT(C2).  The same distinct will be computed twice.  Ideally we would like to compute the distinct once and then split the output.
> I suspect that optimizing the inner plan is more important because there are more situations where this occurs.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.