spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chester Chen <>
Subject Re: spark config params conventions
Date Fri, 14 Mar 2014 15:17:36 GMT
Based on typesafe config maintainer's response, with latest version of typeconfig, the double
quote is no longer needed for key like spark.speculation, so you don't need code to strip
the quotes

Alpine data labs

Sent from my iPhone

On Mar 12, 2014, at 2:50 PM, Aaron Davidson <> wrote:

> One solution for typesafe config is to use
> "spark.speculation" = true
> Typesafe will recognize the key as a string rather than a path, so the name will actually
be "\"spark.speculation\"", so you need to handle this contingency when passing the config
operations to spark (stripping the quotes from the key).
> Solving this in Spark itself is a little tricky because there are ~5 such conflicts (spark.serializer,
spark.speculation, spark.locality.wait, spark.shuffle.spill, and spark.cleaner.ttl), some
of which are used pretty frequently. We could provide aliases for all of these in Spark, but
actually deprecating the old ones would affect many users, so we could only do that if enough
users would benefit from fully hierarchical config options.
> On Wed, Mar 12, 2014 at 9:24 AM, Mark Hamstra <> wrote:
>> That's the whole reason why some of the intended configuration changes were backed
out just before the 0.9.0 release.  It's a well-known issue, even if a completely satisfactory
solution isn't as well-known and is probably something which should do another iteration on.

>> On Wed, Mar 12, 2014 at 9:10 AM, Koert Kuipers <> wrote:
>>> i am reading the spark configuration params from another configuration object
(typesafe config) before setting them as system properties.
>>> i noticed typesafe config has trouble with settings like:
>>> spark.speculation=true
>>> spark.speculation.interval=0.5
>>> the issue seems to be that if spark.speculation is a "container" that has  more
values inside then it cannot be also a value itself, i think. so this would work fine:
>>> spark.speculation.enabled=true
>>> spark.speculation.interval=0.5
>>> just a heads up. i would probably suggest we avoid this situation.

  • Unnamed multipart/alternative (inline, 7-Bit, 0 bytes)
View raw message