spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Or (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SPARK-8873) Support cleaning up shuffle files when using shuffle service in Mesos
Date Thu, 09 Jul 2015 20:29:05 GMT

     [ https://issues.apache.org/jira/browse/SPARK-8873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Andrew Or updated SPARK-8873:
-----------------------------
    Affects Version/s: 1.5.0

> Support cleaning up shuffle files when using shuffle service in Mesos
> ---------------------------------------------------------------------
>
>                 Key: SPARK-8873
>                 URL: https://issues.apache.org/jira/browse/SPARK-8873
>             Project: Spark
>          Issue Type: Improvement
>          Components: Mesos
>    Affects Versions: 1.2.0
>            Reporter: Timothy Chen
>            Priority: Critical
>              Labels: mesos
>
> With dynamic allocation enabled with Mesos, drivers can launch with shuffle data cached
in the external shuffle service.
> However, there is no reliable way to let the shuffle service clean up the shuffle data
when the driver exits, since it may crash before it notifies the shuffle service and shuffle
data will be cached forever.
> We need to implement a reliable way to detect driver termination and clean up shuffle
data accordingly.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message