spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mridul Muralidharan (JIRA)" <>
Subject [jira] [Commented] (SPARK-2277) Make TaskScheduler track whether there's host on a rack
Date Thu, 03 Jul 2014 15:20:24 GMT


Mridul Muralidharan commented on SPARK-2277:

I have not rechecked that the code, but the way it was originally written by me was :

a) Task preference is decoupled from availability of the node.
For example, we need not have an executor on a host for which a block has host preference
(example dfs blocks on a shared cluster)
Also note that a block might have one or more preferred location.

b) We lookup the rack for the preferred location to get preferred rack.
As with (a), there need not be an executor on that rack. This is just the rack preference.

c) At schedule time, for an executor, we lookup the host/rack of the executors location -
and decide appropriately based on that.

In this context, I think your requirement is already handled.
Even if we dont have any hosts alive on a rack, those tasks would still be mentioned with
rack local preference in task set manager.
When an executor comes in (existing or new), we check that executors rack with task preference
- and it would now be marked rack local.

> Make TaskScheduler track whether there's host on a rack
> -------------------------------------------------------
>                 Key: SPARK-2277
>                 URL:
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core
>    Affects Versions: 1.0.0
>            Reporter: Rui Li
> When TaskSetManager adds a pending task, it checks whether the tasks's preferred location
is available. Regarding RACK_LOCAL task, we consider the preferred rack available if such
a rack is defined for the preferred host. This is incorrect as there may be no alive hosts
on that rack at all. Therefore, TaskScheduler should track the hosts on each rack, and provides
an API for TaskSetManager to check if there's host alive on a specific rack.

This message was sent by Atlassian JIRA

View raw message