spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Anirudh Ramanathan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-24028) [K8s] Creating secrets and config maps before creating the driver pod has unpredictable behavior
Date Thu, 19 Apr 2018 22:08:00 GMT

    [ https://issues.apache.org/jira/browse/SPARK-24028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16444881#comment-16444881
] 

Anirudh Ramanathan commented on SPARK-24028:
--------------------------------------------

This is curious. Is this a recent change? You mention a 1.9.4 cluster has this issue. 
Like Yinan said, it doesn't sound like the right behavior if an empty file is found instead
of what you expect - the expectation is that the secret/configmap will exist and be mounted,
or not exist and cause the pod to go pending and retrying till mounting succeeds.

> [K8s] Creating secrets and config maps before creating the driver pod has unpredictable
behavior
> ------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-24028
>                 URL: https://issues.apache.org/jira/browse/SPARK-24028
>             Project: Spark
>          Issue Type: Bug
>          Components: Kubernetes
>    Affects Versions: 2.3.0
>            Reporter: Matt Cheah
>            Priority: Critical
>
> Currently we create the Kubernetes resources the driver depends on - such as the properties
config map and secrets to mount into the pod - only after we create the driver pod. This is
because we want these extra objects to immediately have an owner reference to be tied to the
driver pod.
> On our Kubernetes 1.9.4. cluster, we're seeing that sometimes this works fine, but other
times the driver ends up being started with empty volumes instead of volumes with the contents
of the secrets we expect. The result is that sometimes the driver will start without these
files mounted, which leads to various failures if the driver requires these files to be present
early on in their code. Missing the properties file config map, for example, would mean spark-submit
doesn't have a properties file to read at all. See the warning on [https://kubernetes.io/docs/concepts/storage/volumes/#secret.]
> Unfortunately we cannot link owner references to non-existent objects, so we have to
do this instead:
>  # Create the auxiliary resources without any owner references.
>  # Create the driver pod mounting these resources into volumes, as before.
>  # If #2 fails, clean up the resources created in #1.
>  # Edit the auxiliary resources to have an owner reference for the driver pod.
> The multi-step approach leaves a small chance for us to leak resources - for example,
if we fail to make the resource edits in #4 for some reason. This also changes the permissioning
mode required for spark-submit - credentials provided to spark-submit need to be able to edit
resources in addition to creating them.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message