spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron Davidson (JIRA)" <j...@apache.org>
Subject [jira] [Created] (SPARK-4277) Support external shuffle service on Worker
Date Thu, 06 Nov 2014 18:43:33 GMT
Aaron Davidson created SPARK-4277:
-------------------------------------

             Summary: Support external shuffle service on Worker
                 Key: SPARK-4277
                 URL: https://issues.apache.org/jira/browse/SPARK-4277
             Project: Spark
          Issue Type: Bug
          Components: Deploy, Spark Core
            Reporter: Aaron Davidson
            Assignee: Aaron Davidson


It's less useful to have an external shuffle service on the Spark Standalone Worker than on
YARN or Mesos (as executor allocations tend to be more static), but it would be good to help
test the code path. It would also make Spark more resilient to particular executor failures.

Cool side-feature: When SPARK-4236 is fixed and integrated, the Worker will take care of cleaning
up executor directories, which will mean executors terminate more quickly and that we don't
leak data if the executor dies forcefully.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message