spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Adrian Bridgett (JIRA)" <>
Subject [jira] [Commented] (SPARK-21521) History service requires user is in any group
Date Mon, 24 Jul 2017 20:40:00 GMT


Adrian Bridgett commented on SPARK-21521:

Hmm, I didn't check that (it's actually just a local filesystem in this case, but I take the
point that it's accessed via the Hadoop FS API).

Not sure why it's changed behaviour in 2.2.0 compared to 2.0.2 though. 

Just checked access via hadoop (using hdfs command line) and it's certainly still able to
access it:
~# id
uid=0(root) gid=0(root) groups=0(root),997(airflow)

~# ls -l /var/log/spark/events/a5bab156-b4c2-41a2-93f2-11ba78c99c6e-12221.lz4
-rwxrwx--- 1 ubuntu ubuntu 13982 Jul 24 17:57 /var/log/spark/events/a5bab156-b4c2-41a2-93f2-11ba78c99c6e-12221.lz4

~# hdfs dfs -cat /var/log/spark/events/a5bab156-b4c2-41a2-93f2-11ba78c99c6e-12221.lz4id

> History service requires user is in any group
> ---------------------------------------------
>                 Key: SPARK-21521
>                 URL:
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 2.2.0
>            Reporter: Adrian Bridgett
> (Regression cf. 2.0.2)
> We run spark as several users, these write to the history location where the files are
saved as those users with permissions of 770 (this is hardcoded in EventLoggingListener.scala).
> The history service runs as root so that it has permissions on these files (see
> This worked fine in v2.0.2, however in v2.2.0 the events are being skipped unless I add
the root user into each users group at which point they are seen.
> We currently have all acls configuration unset.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message