reef-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Brian Cho (JIRA)" <>
Subject [jira] [Commented] (REEF-771) Locality-aware Task Scheduler
Date Tue, 05 Jan 2016 02:21:39 GMT


Brian Cho commented on REEF-771:


The Task scheduler works as follows:
1. If there are no queued tasks (each task runs on an evaluator), the number of evaluators
remains at min (*)
2. If there are queued tasks, additional evaluators will be launched

I think for situation 1, we should just do a best-effort locality among the open evaluators.
For situation 2, we could indeed prefer launching an additional evaluator on a local node
to a queued task. Does this make sense?

(*) I just realized that I haven't made the changes to REEF-701. Thus, the current min is
always one. I'll take care of the PR soon.

> Locality-aware Task Scheduler
> -----------------------------
>                 Key: REEF-771
>                 URL:
>             Project: REEF
>          Issue Type: Improvement
>          Components: REEF Examples
>            Reporter: Byung-Gon Chun
> The current TaskScheduler example does not consider data locality when tasks are launched.
A nice extension of the example is to consider data locality to schedule tasks when data's
stored in HDFS. 

This message was sent by Atlassian JIRA

View raw message