spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Anirudh Ramanathan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-24137) [K8s] Mount temporary directories in emptydir volumes
Date Wed, 02 May 2018 05:54:00 GMT

    [ https://issues.apache.org/jira/browse/SPARK-24137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16460554#comment-16460554
] 

Anirudh Ramanathan commented on SPARK-24137:
--------------------------------------------

SGTM. Let's try and share as much code as possible with https://issues.apache.org/jira/browse/SPARK-23529.

> [K8s] Mount temporary directories in emptydir volumes
> -----------------------------------------------------
>
>                 Key: SPARK-24137
>                 URL: https://issues.apache.org/jira/browse/SPARK-24137
>             Project: Spark
>          Issue Type: Bug
>          Components: Kubernetes
>    Affects Versions: 2.3.0
>            Reporter: Matt Cheah
>            Priority: Major
>
> Currently the Spark local directories do not get any volumes and volume mounts, which
means we're writing Spark shuffle and cache contents to the file system mounted by Docker.
This can be terribly inefficient. We should use emptydir volumes for these directories instead
for significant performance improvements.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message