spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From liyinan926 <...@git.apache.org>
Subject [GitHub] spark pull request #20553: [SPARK-23285][K8S] Add a config property for spec...
Date Thu, 08 Feb 2018 22:36:00 GMT
GitHub user liyinan926 opened a pull request:

    https://github.com/apache/spark/pull/20553

    [SPARK-23285][K8S] Add a config property for specifying physical executor cores

    As discussed in SPARK-23285, this PR introduces a new configuation property `spark.kubernetes.executor.cores`
for specifying the phyiscal CPU cores requested for each executor pod. This is to avoid changing
the semantics of `spark.executor.cores` and `spark.task.cpus` and their role in task scheduling,
task parallelism, dynamic resource allocation, etc. The new configuraiton property only determines
the physical CPU cores available to an executor. An executor can still run multiple tasks
simultaneously by using appropriate values for `spark.executor.cores` and `spark.task.cpus`.
    
    ## What changes were proposed in this pull request?
    
    (Please fill in changes proposed in this fix)
    
    ## How was this patch tested?
    
    (Please explain how this patch was tested. E.g. unit tests, integration tests, manual
tests)
    (If this patch involves UI changes, please attach a screenshot; otherwise, remove this)
    
    Please review http://spark.apache.org/contributing.html before opening a pull request.


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/liyinan926/spark-k8s master

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/20553.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #20553
    
----
commit 50ebb5068a35a9a0f2becd27153bdc7cc7aae251
Author: Yinan Li <liyinan926@...>
Date:   2018-02-08T22:22:46Z

    [SPARK-23285][K8S] Add a config property for specifying physical executor cores
    
    As discussed in SPARK-23285, this PR introduces a new configuation property `spark.kubernetes.executor.cores`
for specifying the phyiscal CPU cores requested for each executor pod. This is to avoid changing
the semantics of `spark.executor.cores` and `spark.task.cpus` and their role in task scheduling,
task parallelism, dynamic resource allocation, etc. The new configuraiton property only determines
the physical CPU cores available to an executor. An executor can still run multiple tasks
simultaneously by using appropriate values for `spark.executor.cores` and `spark.task.cpus`.

----


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Mime
View raw message