spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dale Richardson (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SPARK-6214) Allow configuration options to use a simple expression language
Date Sun, 22 Mar 2015 03:49:10 GMT

     [ https://issues.apache.org/jira/browse/SPARK-6214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Dale Richardson updated SPARK-6214:
-----------------------------------
    Description: 
This is a proposal to allow for configuration options to be specified via a simple expression
language.  This language would have the following features:

* Allow for basic arithmetic (+-/*) with support for bracketed expressions and standard precedence
rules.

* Support for and normalisation of common units of scale eg. MB, GB, seconds,minutes,hours,
days and weeks.

* Allow for the referencing of basic environmental information currently defined as:
numCores:             Number of cores assigned to the JVM
physicalMemoryBytes: Memory size of hosting machine 
JVMTotalMemoryBytes: current bytes of memory allocated to the JVM

JVMMaxMemoryBytes:    Maximum number of bytes of memory available to the JVM
JVMFreeMemoryBytes:   maxMemoryBytes - totalMemoryBytes

* Allow for the limited referencing of other configuration values when specifying values.
(Other configuration values must be initialised and explicitly passed into the expression
evaluator for this functionality to be enabled).
 
Such a feature would have the following end-user benefits:
* Allow for the flexibility in specifying time intervals or byte quantities in appropriate
and easy to follow units  e.g. 1 week rather rather then 604800 seconds

* Have a consistent means of entering configuration information regardless of the configuration
option being added. (eg questions such as ‘is the particular option specified in ms or seconds?’
become irrelevant, because the user can pick what ever unit makes sense for the magnitude
of the value they are specifying)

* Allow for the scaling of a configuration option in relation to a system attributes. e.g.
--cores "numCores - 1"
spark.executor.memory = physicalMemoryBytes - 1.5 GB

* Being able to scale multiple configuration options together eg:
spark.driver.memory = 0.75 * physicalMemoryBytes
spark.driver.maxResultSize = spark.driver.memory * 0.8


A safety features has been added so that the expression evaluator is only used if the configuration
strings contains a magic character (currently '!') as the first character.


  was:
This is a proposal to allow for configuration options to be specified via a simple expression
language.  This language would have the following features:

* Allow for basic arithmetic (+-/*) with support for bracketed expressions and standard precedence
rules.

* Support for and normalisation of common units of scale eg. MB, GB, seconds,minutes,hours,
days and weeks.

* Allow for the referencing of basic environmental information currently defined as:
numCores:             Number of cores assigned to the JVM
physicalMemoryBytes: Memory size of hosting machine 
JVMTotalMemoryBytes: current bytes of memory allocated to the JVM

JVMMaxMemoryBytes:    Maximum number of bytes of memory available to the JVM
JVMFreeMemoryBytes:   maxMemoryBytes - totalMemoryBytes

* Allow for the limited referencing of other configuration values when specifying values.
(Other configuration values must be initialised and explicitly passed into the expression
evaluator for this functionality to be enabled).
 
Such a feature would have the following end-user benefits:
* Allow for the flexibility in specifying time intervals or byte quantities in appropriate
and easy to follow units  e.g. 1 week rather rather then 604800 seconds

* Have a consistent means of entering configuration information regardless of the configuration
option being added. (eg questions such as ‘is the particular option specified in ms or seconds?’
become irrelevant, because the user can pick what ever unit makes sense for the magnitude
of the value they are specifying)

* Allow for the scaling of a configuration option in relation to a system attributes. e.g.
--cores "numCores - 1"
spark.executor.memory = physicalMemoryBytes - 1.5 GB

* Being able to scale multiple configuration options together eg:
spark.driver.memory = 0.75 * physicalMemoryBytes
spark.driver.maxResultSize = spark.driver.memory * 0.8



> Allow configuration options to use a simple expression language
> ---------------------------------------------------------------
>
>                 Key: SPARK-6214
>                 URL: https://issues.apache.org/jira/browse/SPARK-6214
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core
>            Reporter: Dale Richardson
>            Priority: Minor
>
> This is a proposal to allow for configuration options to be specified via a simple expression
language.  This language would have the following features:

> * Allow for basic arithmetic (+-/*) with support for bracketed expressions and standard
precedence rules.
> * Support for and normalisation of common units of scale eg. MB, GB, seconds,minutes,hours,
days and weeks.
> * Allow for the referencing of basic environmental information currently defined as:
> numCores:             Number of cores assigned to the JVM
> physicalMemoryBytes: Memory size of hosting machine 
> JVMTotalMemoryBytes: current bytes of memory allocated to the JVM

> JVMMaxMemoryBytes:    Maximum number of bytes of memory available to the JVM
> JVMFreeMemoryBytes:   maxMemoryBytes - totalMemoryBytes
> * Allow for the limited referencing of other configuration values when specifying values.
(Other configuration values must be initialised and explicitly passed into the expression
evaluator for this functionality to be enabled).
>  
> Such a feature would have the following end-user benefits:
> * Allow for the flexibility in specifying time intervals or byte quantities in appropriate
and easy to follow units  e.g. 1 week rather rather then 604800 seconds
> * Have a consistent means of entering configuration information regardless of the configuration
option being added. (eg questions such as ‘is the particular option specified in ms or seconds?’
become irrelevant, because the user can pick what ever unit makes sense for the magnitude
of the value they are specifying)
> * Allow for the scaling of a configuration option in relation to a system attributes.
e.g.
> --cores "numCores - 1"
> spark.executor.memory = physicalMemoryBytes - 1.5 GB
> * Being able to scale multiple configuration options together eg:
> spark.driver.memory = 0.75 * physicalMemoryBytes
> spark.driver.maxResultSize = spark.driver.memory * 0.8
> A safety features has been added so that the expression evaluator is only used if the
configuration strings contains a magic character (currently '!') as the first character.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message