spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matt Cheah (JIRA)" <j...@apache.org>
Subject [jira] [Created] (SPARK-24137) [K8s] Mount temporary directories in emptydir volumes
Date Tue, 01 May 2018 17:01:00 GMT
Matt Cheah created SPARK-24137:
----------------------------------

             Summary: [K8s] Mount temporary directories in emptydir volumes
                 Key: SPARK-24137
                 URL: https://issues.apache.org/jira/browse/SPARK-24137
             Project: Spark
          Issue Type: Bug
          Components: Kubernetes
    Affects Versions: 2.3.0
            Reporter: Matt Cheah


Currently the Spark local directories do not get any volumes and volume mounts, which means
we're writing Spark shuffle and cache contents to the file system mounted by Docker. This
can be terribly inefficient. We should use emptydir volumes for these directories instead
for significant performance improvements.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message