spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eugene Chepurniy <>
Subject Spark executor on Mesos - how to set effective user id?
Date Mon, 19 Oct 2015 12:14:05 GMT
Hi everyone!
While we are trying to utilize Spark On Mesos cluster, we are facing an
issue related to effective linux user id being used to start executors on
Mesos slaves: all executors are trying to use driver's linux user id to
start on Mesos slaves. 
Let me explain in detail: spark driver program (which is going to spawn
Spark on Mesos in coarse mode) is started as unprivileged linux user, for
example 'user1'. We have Spark distribution unpacked and ready-to-use on
every mesos slave (placed at /opt/spark, 'spark.mesos.executor.home' is
pointing to this folder). And after attempt to run every executor fails to
start with error log telling user 'user1' is not available. And it is really
true - there is no 'user1' present on Mesos slaves. 
So my question is: how can I control effective user id which will be used
for start executors on Mesos?
Actually I was trying to setup SPARK_USER=nobody on every slave but it
wasn't useful. 
Thanks for advice if any.

View this message in context:
Sent from the Apache Spark User List mailing list archive at

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message