spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Oz Ben-Ami (JIRA)" <j...@apache.org>
Subject [jira] [Created] (SPARK-23082) Allow separate node selectors for driver and executors in Kubernetes
Date Mon, 15 Jan 2018 17:56:00 GMT
Oz Ben-Ami created SPARK-23082:
----------------------------------

             Summary: Allow separate node selectors for driver and executors in Kubernetes
                 Key: SPARK-23082
                 URL: https://issues.apache.org/jira/browse/SPARK-23082
             Project: Spark
          Issue Type: Improvement
          Components: Kubernetes, Spark Submit
    Affects Versions: 2.2.0, 2.3.0
            Reporter: Oz Ben-Ami


In YARN, we can use spark.yarn.am.nodeLabelExpression to submit the Spark driver to a different
set of nodes from its executors. In Kubernetes, we can specify spark.kubernetes.node.selector.[labelKey],
but we can't use separate options for the driver and executors. This would be useful for the
particular use case where executors can go on more ephemeral nodes (eg, with cluster autoscaling,
or preemptible/spot instances), but the driver should use a more persistent machine.
The required change would be minimal, essentially just using different config keys in [https://github.com/apache/spark/blob/master/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/steps/BasicDriverConfigurationStep.scala#L90] and
[https://github.com/apache/spark/blob/0b2eefb674151a0af64806728b38d9410da552ec/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodFactory.scala#L73] instead
of KUBERNETES_NODE_SELECTOR_PREFIX for both.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message