hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daniel Dai (JIRA)" <j...@apache.org>
Subject [jira] Updated: (PIG-1587) Cloning utility functions for new logical plan
Date Wed, 01 Sep 2010 17:19:56 GMT

     [ https://issues.apache.org/jira/browse/PIG-1587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Daniel Dai updated PIG-1587:
----------------------------

    Description: 
We sometimes need to copy a logical operator/plan when writing an optimization rule. Currently
copy an operator/plan is awkward. We need to write some utilities to facilitate this process.
Swati contribute PIG-1510 but we feel it still cannot address most use cases. I propose to
add some more utilities into new logical plan:

all LogicalExpressions:
{code}
copy(LogicalExpressionPlan newPlan, boolean keepUid);
{code}
* Do a shallow copy of the logical expression operator (except for fieldSchema, uidOnlySchema,
ProjectExpression.attachedRelationalOp)
* Set the plan to newPlan
* If keepUid is true, further copy uidOnlyFieldSchema

all LogicalRelationalOperators:
{code}
copy(LogicalPlan newPlan, boolean keepUid);
{code}
* Do a shallow copy of the logical relational operator (except for schema, uid related fields)
* Set the plan to newPlan;
* If the operator have inner plan/expression plan, copy the whole inner plan with the same
keepUid flag (Especially, LOInnerLoad will copy its inner project, with the same keepUid flag)
* If keepUid is true, further copy uid related fields (LOUnion.uidMapping, LOCogroup.groupKeyUidOnlySchema,
LOCogroup.generatedInputUids)

LogicalExpressionPlan.java
{code}
LogicalExpressionPlan copy(LogicalRelationalOperator attachedRelationalOp, boolean keepUid);
LogicalExpressionPlan copyAbove(LogicalExpression leave, LogicalRelationalOperator attachedRelationalOp,
boolean keepUid);
LogicalExpressionPlan copyBelow(LogicalExpression root, LogicalRelationalOperator attachedRelationalOp,
boolean keepUid);
{code}
* Create a new logical expression plan and copy expression operator along with connection
with the same keepUid flag
* Set all ProjectExpression.attachedRelationalOp to attachedRelationalOp parameter

{code}
Pair<List<Operator>, List<Operator>> merge(LogicalExpressionPlan plan, LogicalRelationalOperator
attachedRelationalOp);
{code}
* Merge plan into the current logical expression plan as an independent tree
* attachedRelationalOp is the destination operator new logical expression plan attached to
* return the sources/sinks of this independent tree


LogicalPlan.java
{code}
LogicalPlan copy(LOForEach foreach, boolean keepUid);
LogicalPlan copyAbove(LogicalRelationalOperator leave, LOForEach foreach, boolean keepUid);
LogicalPlan copyBelow(LogicalRelationalOperator root, LOForEach foreach, boolean keepUid);
{code}
* Main use case to copy inner plan of ForEach
* Create a new logical plan and copy relational operator along with connection
* Copy all expression plans inside relational operator, set plan and attachedRelationalOp
properly
* If the plan is ForEach inner plan, param foreach is the destination ForEach operator; otherwise,
pass null

{code}
Pair<List<Operator>, List<Operator>> merge(LogicalPlan plan, LOForEach foreach);
{code}
* Merge plan into the current logical plan as an independent tree
* foreach is the destination LOForEach is the destination plan is a ForEach inner plan; otherwise,
pass null
* return the sources/sinks of this independent tree


  was:
We sometimes need to copy a logical operator/plan when writing an optimization rule. Currently
copy an operator/plan is awkward. We need to write some utilities to facilitate this process.
Swati contribute PIG-1510 but we feel it still cannot address most use cases. I propose to
add some more utilities into new logical plan:

all LogicalExpressions:
{code}
copy(LogicalExpressionPlan newPlan, boolean keepUid);
{code}
* Do a shallow copy of the logical expression operator (except for fieldSchema, uidOnlySchema,
ProjectExpression.attachedRelationalOp)
* Set the plan to newPlan
* If keepUid is true, further copy uidOnlyFieldSchema

all LogicalRelationalOperators:
{code}
copy(LogicalPlan newPlan, boolean keepUid);
{code}
* Do a shallow copy of the logical relational operator (except for schema, uid related fields)
* Set the plan to newPlan;
* If the operator have inner plan/expression plan, copy the whole inner plan with the same
keepUid flag (Especially, LOInnerLoad will copy its inner project, with the same keepUid flag)
* If keepUid is true, further copy uid related fields (LOUnion.uidMapping, LOCogroup.groupKeyUidOnlySchema,
LOCogroup.generatedInputUids)

LogicalExpressionPlan.java
{code}
LogicalExpressionPlan copy(LogicalRelationalOperator attachedRelationalOp, boolean keepUid);
{code}
* Copy expression operator along with connection with the same keepUid flag
* Set all ProjectExpression.attachedRelationalOp to attachedRelationalOp parameter

{code}
List<Operator> merge(LogicalExpressionPlan plan);
{code}
* Merge plan into the current logical expression plan as an independent tree
* return the sources of this independent tree


LogicalPlan.java
{code}
LogicalPlan copy(boolean keepUid);
{code}
* Main use case to copy inner plan of ForEach
* Copy all relational operator along with connection
* Copy all expression plans inside relational operator, set plan and attachedRelationalOp
properly

{code}
List<Operator> merge(LogicalPlan plan);
{code}
* Merge plan into the current logical plan as an independent tree
* return the sources of this independent tree



> Cloning utility functions for new logical plan
> ----------------------------------------------
>
>                 Key: PIG-1587
>                 URL: https://issues.apache.org/jira/browse/PIG-1587
>             Project: Pig
>          Issue Type: Improvement
>          Components: impl
>    Affects Versions: 0.8.0
>            Reporter: Daniel Dai
>             Fix For: 0.9.0
>
>
> We sometimes need to copy a logical operator/plan when writing an optimization rule.
Currently copy an operator/plan is awkward. We need to write some utilities to facilitate
this process. Swati contribute PIG-1510 but we feel it still cannot address most use cases.
I propose to add some more utilities into new logical plan:
> all LogicalExpressions:
> {code}
> copy(LogicalExpressionPlan newPlan, boolean keepUid);
> {code}
> * Do a shallow copy of the logical expression operator (except for fieldSchema, uidOnlySchema,
ProjectExpression.attachedRelationalOp)
> * Set the plan to newPlan
> * If keepUid is true, further copy uidOnlyFieldSchema
> all LogicalRelationalOperators:
> {code}
> copy(LogicalPlan newPlan, boolean keepUid);
> {code}
> * Do a shallow copy of the logical relational operator (except for schema, uid related
fields)
> * Set the plan to newPlan;
> * If the operator have inner plan/expression plan, copy the whole inner plan with the
same keepUid flag (Especially, LOInnerLoad will copy its inner project, with the same keepUid
flag)
> * If keepUid is true, further copy uid related fields (LOUnion.uidMapping, LOCogroup.groupKeyUidOnlySchema,
LOCogroup.generatedInputUids)
> LogicalExpressionPlan.java
> {code}
> LogicalExpressionPlan copy(LogicalRelationalOperator attachedRelationalOp, boolean keepUid);
> LogicalExpressionPlan copyAbove(LogicalExpression leave, LogicalRelationalOperator attachedRelationalOp,
boolean keepUid);
> LogicalExpressionPlan copyBelow(LogicalExpression root, LogicalRelationalOperator attachedRelationalOp,
boolean keepUid);
> {code}
> * Create a new logical expression plan and copy expression operator along with connection
with the same keepUid flag
> * Set all ProjectExpression.attachedRelationalOp to attachedRelationalOp parameter
> {code}
> Pair<List<Operator>, List<Operator>> merge(LogicalExpressionPlan plan,
LogicalRelationalOperator attachedRelationalOp);
> {code}
> * Merge plan into the current logical expression plan as an independent tree
> * attachedRelationalOp is the destination operator new logical expression plan attached
to
> * return the sources/sinks of this independent tree
> LogicalPlan.java
> {code}
> LogicalPlan copy(LOForEach foreach, boolean keepUid);
> LogicalPlan copyAbove(LogicalRelationalOperator leave, LOForEach foreach, boolean keepUid);
> LogicalPlan copyBelow(LogicalRelationalOperator root, LOForEach foreach, boolean keepUid);
> {code}
> * Main use case to copy inner plan of ForEach
> * Create a new logical plan and copy relational operator along with connection
> * Copy all expression plans inside relational operator, set plan and attachedRelationalOp
properly
> * If the plan is ForEach inner plan, param foreach is the destination ForEach operator;
otherwise, pass null
> {code}
> Pair<List<Operator>, List<Operator>> merge(LogicalPlan plan, LOForEach
foreach);
> {code}
> * Merge plan into the current logical plan as an independent tree
> * foreach is the destination LOForEach is the destination plan is a ForEach inner plan;
otherwise, pass null
> * return the sources/sinks of this independent tree

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message