spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jiang Xingbo (JIRA)" <j...@apache.org>
Subject [jira] [Created] (SPARK-24340) Clean up non-shuffle disk block manager files following executor death
Date Tue, 22 May 2018 07:12:00 GMT
Jiang Xingbo created SPARK-24340:
------------------------------------

             Summary: Clean up non-shuffle disk block manager files following executor death
                 Key: SPARK-24340
                 URL: https://issues.apache.org/jira/browse/SPARK-24340
             Project: Spark
          Issue Type: Improvement
          Components: Spark Core
    Affects Versions: 2.4.0
            Reporter: Jiang Xingbo


Currently we only clean up local folders on application removed, and we don't clean up non-shuffle
files, such as temp. shuffle blocks, cached RDD/broadcast blocks, spill files, etc. and this
can cause disk space leaks when executors periodically die and are replaced.

To avoid this source of disk space leak, we can clean up executor disk store files except
for shuffle index and data files on executor finished.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message