spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From maropu <...@git.apache.org>
Subject [GitHub] spark pull request #19440: [SPARK-21871][SQL] Fix infinite loop when bytecod...
Date Fri, 06 Oct 2017 01:36:41 GMT
Github user maropu commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19440#discussion_r143094785
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/WholeStageCodegenExec.scala
---
    @@ -392,12 +392,16 @@ case class WholeStageCodegenExec(child: SparkPlan) extends UnaryExecNode
with Co
     
         // Check if compiled code has a too large function
         if (maxCodeSize > sqlContext.conf.hugeMethodLimit) {
    -      logWarning(s"Found too long generated codes and JIT optimization might not work:
" +
    -        s"the bytecode size was $maxCodeSize, this value went over the limit " +
    +      logInfo(s"Found too long generated codes and JIT optimization might not work: "
+
    +        s"the bytecode size ($maxCodeSize) is above the limit " +
             s"${sqlContext.conf.hugeMethodLimit}, and the whole-stage codegen was disabled
" +
             s"for this plan. To avoid this, you can raise the limit " +
    -        s"${SQLConf.WHOLESTAGE_HUGE_METHOD_LIMIT.key}:\n$treeString")
    -      return child.execute()
    +        s"`${SQLConf.WHOLESTAGE_HUGE_METHOD_LIMIT.key}`:\n$treeString")
    +      child match {
    +        // For batch file source scan, we should continue executing it
    +        case f: FileSourceScanExec if f.supportsBatch => // do nothing
    --- End diff --
    
    yea, I totally agree that we need to refactor this in future. Anyway, it's ok for now.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Mime
View raw message