spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Imran Rashid (JIRA)" <>
Subject [jira] [Created] (SPARK-24016) Yarn does not update node blacklist in static allocation
Date Wed, 18 Apr 2018 15:45:00 GMT
Imran Rashid created SPARK-24016:

             Summary: Yarn does not update node blacklist in static allocation
                 Key: SPARK-24016
             Project: Spark
          Issue Type: Improvement
          Components: Scheduler, YARN
    Affects Versions: 2.3.0
            Reporter: Imran Rashid

Task-based blacklisting keeps track of bad nodes, and updates YARN with that set of nodes
so that Spark will not receive more containers on that node.  However, that only happens with
dynamic allocation.  Though its far more important with dynamic allocation, even with static
allocation this matters; if executors die, or if the cluster was too busy at the original
resource request to give all the containers, the spark application will add new containers
in the middle.  And we want an updated node blacklist for that.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message