spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Imran Rashid (JIRA)" <j...@apache.org>
Subject [jira] [Created] (SPARK-24016) Yarn does not update node blacklist in static allocation
Date Wed, 18 Apr 2018 15:45:00 GMT
Imran Rashid created SPARK-24016:
------------------------------------

             Summary: Yarn does not update node blacklist in static allocation
                 Key: SPARK-24016
                 URL: https://issues.apache.org/jira/browse/SPARK-24016
             Project: Spark
          Issue Type: Improvement
          Components: Scheduler, YARN
    Affects Versions: 2.3.0
            Reporter: Imran Rashid


Task-based blacklisting keeps track of bad nodes, and updates YARN with that set of nodes
so that Spark will not receive more containers on that node.  However, that only happens with
dynamic allocation.  Though its far more important with dynamic allocation, even with static
allocation this matters; if executors die, or if the cluster was too busy at the original
resource request to give all the containers, the spark application will add new containers
in the middle.  And we want an updated node blacklist for that.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message