spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Maxim Gekk (JIRA)" <>
Subject [jira] [Created] (SPARK-24761) Check modifiability of config parameters
Date Sun, 08 Jul 2018 19:26:00 GMT
Maxim Gekk created SPARK-24761:

             Summary: Check modifiability of config parameters
                 Key: SPARK-24761
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 2.3.1
            Reporter: Maxim Gekk

Our customers and support team continuously face to the situation when setting a config parameter
via *spark.conf.set()* does not may any effects. It is not clear from parameter's name is
it static parameter or one of the parameter that can be set at runtime for current session
state. It would be useful to have a method of *RuntimeConfig* which could tell to an user
- does the given parameter may effect on the current behavior if he/she change it in the spark-shell
or running notebook. The method can have the following signature:

def isModifiable(key: String): Boolean

Any config parameter can be checked by using the syntax like this:

scala> spark.conf.isModifiable("spark.sql.sources.schemaStringLengthThreshold")
res0: Boolean = false
or for Spark Core parameter:
scala> spark.conf.isModifiable("spark.task.cpus")
res1: Boolean = false

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message