mesos-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jongyoul Lee <jongy...@gmail.com>
Subject Spark-job error on writing result into hadoop w/ switch_user=false
Date Wed, 20 Aug 2014 17:02:28 GMT
Hi,

I've used hdfs 2.3.0-cdh5.0.1, mesos 0.19.1 and spark 1.0.2 that is
re-compiled.

For a security reason, we run hdfs and mesos as hdfs, that is an account
name and not in a root group, and non-root user submit a spark job on
mesos. With no-switch_user, simple job, which only read data from hdfs,
works fine, but when i code more complex job including map and groupBy,
spark job always fails.

When a spark job is running, it create temporary file into output dir on
hdfs and the directory owner is hdfs - same account running mesos -.
Finally, id to sumbit job fails to delete temporary file because of
permission error.

Please advise to solve this problem.

JL

Mime
View raw message