Mailing-List: contact dev-help@gearpump.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@gearpump.incubator.apache.org
Date: Sat, 21 May 2016 12:17:12 +0000 (UTC)
From: "Karol Brejna (JIRA)" <jira@apache.org>
To: dev@gearpump.incubator.apache.org
Message-ID: <JIRA.12971783.1463832679000.258091.1463833032815@Atlassian.JIRA>
In-Reply-To: <JIRA.12971783.1463832679000@Atlassian.JIRA>
References: <JIRA.12971783.1463832679000@Atlassian.JIRA> <JIRA.12971783.1463832679131@arcas>
Subject: [jira] [Commented] (GEARPUMP-149) Binding tasks to workers
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Sat, 21 May 2016 12:17:16 -0000


    [ https://issues.apache.org/jira/browse/GEARPUMP-149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15294986#comment-15294986 ] 

Karol Brejna commented on GEARPUMP-149:
---------------------------------------

Another rationale: 
I've created a cluster - master and worker on PC (default config), a worker on raspi (less memory, less slots). 
When deploying a DAG, the master "avoids" creating an executor on raspi. It prefers "stronger" worker. 
If we had "binding" mechanism, we could force the task to be ran on raspi.

> Binding tasks to workers
> ------------------------
>
>                 Key: GEARPUMP-149
>                 URL: https://issues.apache.org/jira/browse/GEARPUMP-149
>             Project: Apache Gearpump
>          Issue Type: Improvement
>            Reporter: Karol Brejna
>            Priority: Minor
>
> Let's assume that we have a cluster with some "specialized" nodes (for example a worker on IoT device that is responsible for controlling some actuators, dedicated workers to achieve data locality, some hardware specific - mem/cpu - tasks, etc.).
> There should be a way to "hint" Gearpump to prefer some nodes (workers) over others for certain tasks (executors). 
> We already have "localities" feature. Developer can add a configuration (when submitting  the application) to bind a task with a worker:
> {code}
> gearpump {
>   streaming {
>     localities {
>       app1: {            //This is the application name to be submitted
>         workerId: [    //The worker id where to launch Task0_0 and Task0_1
>           TaskId(0,0), TaskId(0,1)
>         ]
>       }
>     }
>   }
> }
> {code}
> The problem is: users have no idea what each Task's task ID is (especially if there are lots of nodes in a DAG).
> We could introduce more flexible "binding" mechanism, for example labeling tasks and workers or using some other "awareness" algorithm.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)