spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Marco Gaido (JIRA)" <j...@apache.org>
Subject [jira] [Created] (SPARK-22215) Add a configuration parameter to set max size for generated classes
Date Fri, 06 Oct 2017 13:57:02 GMT
Marco Gaido created SPARK-22215:
-----------------------------------

             Summary: Add a configuration parameter to set max size for generated classes
                 Key: SPARK-22215
                 URL: https://issues.apache.org/jira/browse/SPARK-22215
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 2.2.0
            Reporter: Marco Gaido


SPARK-18016 introduced an arbitrary threshold for the size of a generated class (https://github.com/apache/spark/blob/83488cc3180ca18f829516f550766efb3095881e/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala#L286).
This value is hardcoded.

Since this is just a guess, in some cases making it smaller can help avoiding the error of
exceeding the maximum number of entries in the Constant Pool.

Then, I suggest to introduce a new configuration parameter, which defaults to the previous
value, but it allows to set this to a smaller one if needed.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message