spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Apache Spark (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-24248) [K8S] Use the Kubernetes cluster as the backing store for the state of pods
Date Fri, 18 May 2018 21:26:00 GMT

    [ https://issues.apache.org/jira/browse/SPARK-24248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16481217#comment-16481217
] 

Apache Spark commented on SPARK-24248:
--------------------------------------

User 'mccheah' has created a pull request for this issue:
https://github.com/apache/spark/pull/21366

> [K8S] Use the Kubernetes cluster as the backing store for the state of pods
> ---------------------------------------------------------------------------
>
>                 Key: SPARK-24248
>                 URL: https://issues.apache.org/jira/browse/SPARK-24248
>             Project: Spark
>          Issue Type: Improvement
>          Components: Kubernetes
>    Affects Versions: 2.3.0
>            Reporter: Matt Cheah
>            Priority: Major
>
> We have a number of places in KubernetesClusterSchedulerBackend right now that maintains
the state of pods in memory. However, the Kubernetes API can always give us the most up to
date and correct view of what our executors are doing. We should consider moving away from
in-memory state as much as can in favor of using the Kubernetes cluster as the source of truth
for pod status. Maintaining less state in memory makes it so that there's a lower chance that
we accidentally miss updating one of these data structures and breaking the lifecycle of executors.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message