spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thomas Graves (JIRA)" <>
Subject [jira] [Commented] (SPARK-27005) Design sketch: Accelerator-aware scheduling
Date Fri, 01 Mar 2019 14:39:00 GMT


Thomas Graves commented on SPARK-27005:

It seems like we are mixing gpu's as static resource vs generic one as accelerators.  Perhaps
we should consider making it generic but then having kind of a #define for gpu's so it can
be consistent for users but if they want other generic ones they work as well.

So for instance of calling it *spark.task.gpus* we could call it spark.task.accelerator.[resource] 
to make that part generic and then we could define gpu as a known type so that its easier
for users ot use the same name there.

I also assume in a few of the places you say search for gpu requirements you mean search for
generic accelerator requirements - like in the *Expand RDD/Stage to support GPU* section?

If you are using a separate queue for the tasks with accelerator needs, does it still go through
locality checks?  Can you expand upon exactly where the queue is you are proposing adding?  
The scheduler goes through the resourceOffer code and down into resourceOfferSingleTaskSet
for each locality level and then into the dequeueTask where it finally pulls from a queue,
is this dequeueTask where you are proposing adding a new queue?

> Design sketch: Accelerator-aware scheduling
> -------------------------------------------
>                 Key: SPARK-27005
>                 URL:
>             Project: Spark
>          Issue Type: Story
>          Components: Spark Core
>    Affects Versions: 3.0.0
>            Reporter: Xingbo Jiang
>            Priority: Major
> This task is to outline a design sketch for the accelerator-aware scheduling SPIP discussion.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message