spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <>
Subject [GitHub] [spark] hvanhovell commented on a change in pull request #32060: [SPARK-34916][SQL] Add condition lambda and rule id to the transform family for early stopping
Date Wed, 07 Apr 2021 08:11:47 GMT

hvanhovell commented on a change in pull request #32060:

File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/QueryPlan.scala
@@ -80,28 +105,57 @@ abstract class QueryPlan[PlanType <: QueryPlan[PlanType]]
    * transformExpressionsDown or transformExpressionsUp should be used.
    * @param rule the rule to be applied to every expression in this operator.
+   * @param cond  a Lambda expression to prune tree traversals. If `cond.apply` returns false
+   *              on an expression T, skips processing T and its subtree; otherwise, processes
+   *              T and its subtree recursively.
+   * @param ruleId is a unique Id for `rule` to prune unnecessary tree traversals. When it
+   *        RuleId.UnknownId, no pruning happens. Otherwise, if `rule`(with id `ruleId`)
has been
+   *        marked as in effective on an expression T, skips processing T and its subtree.
Do not
+   *        pass it if the rule is not purely functional and reads a varying initial state
+   *        different invocations.
-  def transformExpressions(rule: PartialFunction[Expression, Expression]): this.type = {
-    transformExpressionsDown(rule)
+  def transformExpressions(rule: PartialFunction[Expression, Expression],
+    cond: TreePatternBits => Boolean = AlwaysProcess.fn,

Review comment:
       Can we have helpers for this? It is a bit annoying to write place holder syntax all
the time. I think the most common one should be contains(..) right?

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message