spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From gatorsmile <...@git.apache.org>
Subject [GitHub] spark pull request #19480: [SPARK-22226][SQL] splitExpression can create too...
Date Fri, 13 Oct 2017 02:20:58 GMT
Github user gatorsmile commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19480#discussion_r144453024
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
---
    @@ -798,10 +831,46 @@ class CodegenContext {
                |  ${makeSplitFunction(body)}
                |}
              """.stripMargin
    -        addNewFunction(name, code)
    +        addNewFunctionInternal(name, code, inlineToOuterClass = false)
           }
     
    -      foldFunctions(functions.map(name => s"$name(${arguments.map(_._2).mkString(",
")})"))
    +      // Here we store all the methods which have been added to the outer class.
    +      val outerClassFunctions = functions
    +        .filter(_.subclassName.isEmpty)
    +        .map(_.functionName)
    +
    +      // Here we handle all the methods which have been added to the nested subclasses
and
    +      // not to the outer class.
    +      // Since they can be many, their direct invocation in the outer class adds many
entries
    +      // to the outer class' constant pool. This can cause the constant pool to past
JVM limit.
    +      // To avoid this problem, we group them and we call only the grouping methods in
the
    +      // outer class.
    +      val innerClassFunctions = functions
    +        .filter(_.subclassName.isDefined)
    +        .foldLeft(ListMap.empty[(String, String), Seq[String]]) { case (acc, f) =>
    +          val key = (f.subclassName.get, f.subclassInstance.get)
    +          acc.updated(key, acc.getOrElse(key, Seq.empty[String]) ++ Seq(f.functionName))
    +        }
    +        .flatMap { case ((subclassName, subclassInstance), subclassFunctions) =>
    +          if (subclassFunctions.size > CodeGenerator.MERGE_SPLIT_METHODS_THRESHOLD)
{
    +            // Adding a new function to each subclass which contains
    +            // the invocation of all the ones which have been added to
    +            // that subclass
    +            val code = s"""
    +                |private $returnType $func($argString) {
    +                |  ${makeSplitFunction(foldFunctions(subclassFunctions.map(name =>
    +                      s"$name(${arguments.map(_._2).mkString(", ")})")))}
    --- End diff --
    
    missing `|`


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Mime
View raw message