spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stavros Kontopoulos (JIRA)" <j...@apache.org>
Subject [jira] [Created] (SPARK-23857) In mesos cluster mode keytab Spark submit requires the keytab to be available on the local file system.
Date Tue, 03 Apr 2018 14:50:00 GMT
Stavros Kontopoulos created SPARK-23857:
-------------------------------------------

             Summary: In mesos cluster mode keytab Spark submit requires the keytab to be
available on the local file system.
                 Key: SPARK-23857
                 URL: https://issues.apache.org/jira/browse/SPARK-23857
             Project: Spark
          Issue Type: Bug
          Components: Mesos
    Affects Versions: 2.3.0
            Reporter: Stavros Kontopoulos


Users could submit their jobs from an external to the cluster host which may not have the
required keytab locally.  Moreover, in cluster mode it does not make much sense to reference
a local resource unless this is uploaded/stored somewhere in the cluster. For yarn HDFS is
used, on mesos and certainly on DC/OS right now the secret store is used for storing secrets
and consequently keytabs. There is a check [here|https://github.com/apache/spark/blob/7cf9fab33457ccc9b2d548f15dd5700d5e8d08ef/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala#L387] that
makes spark submit difficult to use in such deployment scenarios. On DC/OS the workaround
is to directly submit to the mesos dispatcher rest api by passing the spark.yarn.tab property
pointing to a path within that container where the keytab will be mounted after its fetched
from the secret store. Target is to allow spark submit be flexible enough for mesos in cluster
mode, as DC/OS users often want to deploy using that.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message