spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Owen (JIRA)" <>
Subject [jira] [Commented] (SPARK-9636) Treat $SPARK_HOME as write-only
Date Wed, 05 Aug 2015 10:28:04 GMT


Sean Owen commented on SPARK-9636:

I'm not sure those are as obvious as defaults, or necessarily have write permission either.
Isn't the solution that {{SPARK_LOG_DIR}} should be set if needed?

> Treat $SPARK_HOME as write-only
> -------------------------------
>                 Key: SPARK-9636
>                 URL:
>             Project: Spark
>          Issue Type: Bug
>          Components: Input/Output
>    Affects Versions: 1.4.1
>         Environment: Linux
>            Reporter: Philipp Angerer
>              Labels: easyfix
> when starting spark scripts as user and it is installed in a directory the user has no
write permissions on, many things work fine, except for the logs (e.g. for {{}})
> logs are per default written to {{$SPARK_LOG_DIR}} or (if unset) to {{$SPARK_HOME/logs}}.
> if installed in this way, it should, instead of throwing an error, write logs to {{/var/log/spark/}}.
that’s easy to fix by simply testing a few log dirs in sequence for writability before trying
to use one. i suggest using {{$SPARK_LOG_DIR}} (if set) → {{/var/log/spark/}} → {{~/.cache/spark-logs/}}
→ {{$SPARK_HOME/logs/}}

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message