spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lenin (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SPARK-24105) Spark 2.3.0 on kubernetes
Date Thu, 26 Apr 2018 18:29:00 GMT

     [ https://issues.apache.org/jira/browse/SPARK-24105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Lenin updated SPARK-24105:
--------------------------
    Description: 
Right now its only possible to define node selector configurations thruspark.kubernetes.node.selector.[labelKey].
This gets used for both driver & executor pods. Without the capability to isolate driver
& executor pods, the cluster can run into a livelock scenario, where if there are a
lot of spark submits, can cause the driver pods to fill up the cluster capacity, with no
room for executor pods to do any work.

 

To avoid this deadlock, its required to support node selector (in future affinity/anti-affinity)
configruation by driver & executor.

 

  was:
Right now its only possible to define node selector configurations thruspark.kubernetes.node.selector.[labelKey].
This gets used for both driver & executor pods. Without the capability to isolate driver
& executor pods, the cluster can run into a deadlock scenario, where if there are a lot
of spark submits, can cause the driver pods to fill up the cluster capacity, with no room
for executor pods to do any work.

 

To avoid this deadlock, its required to support node selector (in future affinity/anti-affinity)
configruation by driver & executor.

 


> Spark 2.3.0 on kubernetes
> -------------------------
>
>                 Key: SPARK-24105
>                 URL: https://issues.apache.org/jira/browse/SPARK-24105
>             Project: Spark
>          Issue Type: Improvement
>          Components: Kubernetes
>    Affects Versions: 2.3.0
>            Reporter: Lenin
>            Priority: Major
>
> Right now its only possible to define node selector configurations thruspark.kubernetes.node.selector.[labelKey].
This gets used for both driver & executor pods. Without the capability to isolate driver
& executor pods, the cluster can run into a livelock scenario, where if there are a
lot of spark submits, can cause the driver pods to fill up the cluster capacity, with no
room for executor pods to do any work.
>  
> To avoid this deadlock, its required to support node selector (in future affinity/anti-affinity)
configruation by driver & executor.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message