spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [spark] tgravescs commented on a change in pull request #30504: [SPARK-33544][SQL] Optimizer should not insert filter when explode with CreateArray/CreateMap
Date Tue, 01 Dec 2020 15:44:39 GMT

tgravescs commented on a change in pull request #30504:
URL: https://github.com/apache/spark/pull/30504#discussion_r533514274



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
##########
@@ -48,6 +48,9 @@ object ConstantFolding extends Rule[LogicalPlan] {
       // object and running eval unnecessarily.
       case l: Literal => l
 
+      case Size(c: CreateArray, _) => Literal(c.children.length)
+      case Size(c: CreateMap, _) => Literal(c.children.length / 2)

Review comment:
       so the problem with this is it doesn't work with nested.. like inline with array of
struct:
   
   f(CreateArray(Seq(CreateStruct(Seq('c1)))))
   
   Unless we want to recurse down. I don't think there is any one type we can use for the
complex type generators though, thoughts?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Mime
View raw message