spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From cloud-fan <...@git.apache.org>
Subject [GitHub] spark pull request #20139: [SPARK-22944][SQL] improve FoldablePropagation
Date Wed, 03 Jan 2018 16:45:42 GMT
Github user cloud-fan commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20139#discussion_r159471147
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
---
    @@ -530,38 +533,50 @@ object FoldablePropagation extends Rule[LogicalPlan] {
         if (foldableMap.isEmpty) {
           plan
         } else {
    -      var stop = false
           CleanupAliases(plan.transformUp {
    -        // A leaf node should not stop the folding process (note that we are traversing
up the
    -        // tree, starting at the leaf nodes); so we are allowing it.
    -        case l: LeafNode =>
    -          l
    -
             // We can only propagate foldables for a subset of unary nodes.
    -        case u: UnaryNode if !stop && canPropagateFoldables(u) =>
    +        case u: UnaryNode if canPropagateFoldables(u) =>
               u.transformExpressions(replaceFoldable)
     
    -        // Allow inner joins. We do not allow outer join, although its output attributes
are
    -        // derived from its children, they are actually different attributes: the output
of outer
    -        // join is not always picked from its children, but can also be null.
    +        // Join derives the output attributes from its child while they are actually
not the
    +        // same attributes. For example, the output of outer join is not always picked
from its
    +        // children, but can also be null. We should exclude these miss-derived attributes
when
    +        // propagating the foldable expressions.
             // TODO(cloud-fan): It seems more reasonable to use new attributes as the output
attributes
             // of outer join.
    -        case j @ Join(_, _, Inner, _) if !stop =>
    -          j.transformExpressions(replaceFoldable)
    +        case j @ Join(left, right, joinType, _) =>
    +          val newJoin = j.transformExpressions(replaceFoldable)
    +          val missDerivedAttrsSet: AttributeSet = AttributeSet(joinType match {
    --- End diff --
    
    I'd like to keep the word `miss` as it's really a mistake. We should fix it eventually.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Mime
View raw message