spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Timothy Chen (JIRA)" <j...@apache.org>
Subject [jira] [Created] (SPARK-8873) Support cleaning up shuffle files for drivers launched with Mesos
Date Tue, 07 Jul 2015 16:58:05 GMT
Timothy Chen created SPARK-8873:
-----------------------------------

             Summary: Support cleaning up shuffle files for drivers launched with Mesos
                 Key: SPARK-8873
                 URL: https://issues.apache.org/jira/browse/SPARK-8873
             Project: Spark
          Issue Type: Improvement
            Reporter: Timothy Chen


With dynamic allocation enabled with Mesos, drivers can launch with shuffle data cached in
the external shuffle service.
However, there is no reliable way to let the shuffle service clean up the shuffle data when
the driver exits, since it may crash before it notifies the shuffle service and shuffle data
will be cached forever.
We need to implement a reliable way to detect driver termination and clean up shuffle data
accordingly.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message