gearpump-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Zhong (JIRA)" <>
Subject [jira] [Commented] (GEARPUMP-149) Binding tasks to workers
Date Fri, 27 May 2016 06:10:12 GMT


Sean Zhong commented on GEARPUMP-149:

The task id is stable as long as the DAG is the same, no many how many times you submitted
it as an application. So may be we can develop to tool to generate the task location config
on behalf of the user.

> Binding tasks to workers
> ------------------------
>                 Key: GEARPUMP-149
>                 URL:
>             Project: Apache Gearpump
>          Issue Type: Improvement
>            Reporter: Karol Brejna
>            Priority: Minor
> Let's assume that we have a cluster with some "specialized" nodes (for example a worker
on IoT device that is responsible for controlling some actuators, dedicated workers to achieve
data locality, some hardware specific - mem/cpu - tasks, etc.).
> There should be a way to "hint" Gearpump to prefer some nodes (workers) over others for
certain tasks (executors). 
> We already have "localities" feature. Developer can add a configuration (when submitting
 the application) to bind a task with a worker:
> {code}
> gearpump {
>   streaming {
>     localities {
>       app1: {            //This is the application name to be submitted
>         workerId: [    //The worker id where to launch Task0_0 and Task0_1
>           TaskId(0,0), TaskId(0,1)
>         ]
>       }
>     }
>   }
> }
> {code}
> The problem is: users have no idea what each Task's task ID is (especially if there are
lots of nodes in a DAG).
> We could introduce more flexible "binding" mechanism, for example labeling tasks and
workers or using some other "awareness" algorithm.

This message was sent by Atlassian JIRA

View raw message