spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From gatorsmile <...@git.apache.org>
Subject [GitHub] spark pull request #19480: [SPARK-22226][SQL] splitExpression can create too...
Date Thu, 26 Oct 2017 07:26:54 GMT
Github user gatorsmile commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19480#discussion_r147062715
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
---
    @@ -801,10 +832,49 @@ class CodegenContext {
                |  ${makeSplitFunction(body)}
                |}
              """.stripMargin
    -        addNewFunction(name, code)
    +        addNewFunctionInternal(name, code, inlineToOuterClass = false)
           }
     
    -      foldFunctions(functions.map(name => s"$name(${arguments.map(_._2).mkString(",
")})"))
    +      // Here we store all the methods which have been added to the outer class.
    +      val outerClassFunctions = functions
    +        .filter(_.subclassName.isEmpty)
    +        .map(_.functionName)
    +
    +      val argsString = arguments.map(_._2).mkString(", ")
    +
    +      // Here we handle all the methods which have been added to the nested subclasses
and
    +      // not to the outer class.
    +      // Since they can be many, their direct invocation in the outer class adds many
entries
    +      // to the outer class' constant pool. This can cause the constant pool to past
JVM limit.
    +      // To avoid this problem, we group them and we call only the grouping methods in
the
    +      // outer class.
    +      val innerClassToFunctions = functions
    +        .filter(_.subclassName.isDefined)
    +        .foldLeft(ListMap.empty[(String, String), Seq[String]]) { case (acc, f) =>
    +          val key = (f.subclassName.get, f.subclassInstance.get)
    +          acc.updated(key, acc.getOrElse(key, Seq.empty[String]) ++ Seq(f.functionName))
    --- End diff --
    
    To improve the readability, let us avoid using `foldLeft` here. You can use a mutable
map. 


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Mime
View raw message