pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pi Song (JIRA)" <j...@apache.org>
Subject [jira] Commented: (PIG-273) Need to optimize the ways splits are handled, both in the top level plan and in nested plans.
Date Wed, 18 Jun 2008 12:20:45 GMT

    [ https://issues.apache.org/jira/browse/PIG-273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12605923#action_12605923

Pi Song commented on PIG-273:

I assume this is the next step after pipeline rework.

We can keep inner plans as DAGs so that we still maintain all the knowledge needed for optimization
(One might say we can store inner plans separately and merge them but I think that's more

Idea on how to optimize will come soon!

> Need to optimize the ways splits are handled, both in the top level plan and in nested
> ---------------------------------------------------------------------------------------------
>                 Key: PIG-273
>                 URL: https://issues.apache.org/jira/browse/PIG-273
>             Project: Pig
>          Issue Type: Improvement
>          Components: impl
>            Reporter: Alan Gates
>            Priority: Minor
> Currently, in the new pipeline rework (see PIG-157), splits in the data flow are not
handled efficiently.  
> In the top level plans splits cause all the output data to be written to hdfs and then
reread by each leg of the split.  This forces both a read/write and a new map/reduce pass
when it is not always necessary.  For example, consider:
> A = load 'myfile';
> split A into B  if $0 < 100, C if $0 >= 100;
> B1 = group B by $0;
> ...
> C1 = group B by $1;
> ...
> In this case A will be loaded, and then immediately stored again.  Then a plan will be
executed that handles the B* part of the script, and then another executed that will handle
the C* part of the script.
> In nested plans, each projection of the generate is computed separately, even if they
share common steps in the plan.  For example:
> B = group A by $0;
> C= foreach B {
>     C1 = distinct $1;
>     C2 = filter C1 by $1 > 0;
>     generate group, COUNT(C1), COUNT(C2);
> }
> That will currently be executed with two nested plans, distinct->COUNT(C1) and distinct->filter->COUNT(C2).
 The same distinct will be computed twice.  Ideally we would like to compute the distinct
once and then split the output.
> I suspect that optimizing the inner plan is more important because there are more situations
where this occurs.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message