spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thomas Graves (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-3684) Can't configure local dirs in Yarn mode
Date Tue, 28 Jul 2015 13:21:05 GMT

    [ https://issues.apache.org/jira/browse/SPARK-3684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14644359#comment-14644359
] 

Thomas Graves commented on SPARK-3684:
--------------------------------------

What is thie use case for this?  YARN specifically doesn't allow users to do this so they
don't use a directory they shouldn't be and it doesn't get cleaned up or they run out of space,
etc.  As mentioned you should configure YARN to have what directories are available and then
Spark use them.



> Can't configure local dirs in Yarn mode
> ---------------------------------------
>
>                 Key: SPARK-3684
>                 URL: https://issues.apache.org/jira/browse/SPARK-3684
>             Project: Spark
>          Issue Type: Bug
>          Components: YARN
>    Affects Versions: 1.1.0
>            Reporter: Andrew Or
>
> We can't set SPARK_LOCAL_DIRS or spark.local.dirs because they're not picked up in Yarn
mode. However, we can't set YARN_LOCAL_DIRS or LOCAL_DIRS either because these are overridden
by Yarn.
> I'm trying to set these through SPARK_YARN_USER_ENV. I'm aware that the default behavior
is for Spark to use Yarn's local dirs, but right now there's no way to change it even if the
user wants to.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message