gearpump-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Karol Brejna (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (GEARPUMP-149) Binding tasks to workers
Date Sat, 21 May 2016 12:13:12 GMT

     [ https://issues.apache.org/jira/browse/GEARPUMP-149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Karol Brejna updated GEARPUMP-149:
----------------------------------
    Description: 
Let's assume that we have a cluster with some "specialized" nodes (for example a worker on
IoT device that is responsible for controlling some actuators, dedicated workers to achieve
data locality, some hardware specific - mem/cpu - tasks, etc.).

There should be a way to "hint" Gearpump to prefer some nodes (workers) over others for certain
tasks (executors). 

We already have "localities" feature. Developer can add a configuration (when submitting 
the application) to bind a task with a worker:
{code}
gearpump {
  streaming {
    localities {
      app1: {            //This is the application name to be submitted
        workerId: [    //The worker id where to launch Task0_0 and Task0_1
          TaskId(0,0), TaskId(0,1)
        ]
      }
    }
  }
}
{code}

The problem is: users have no idea what each Task's task ID is (especially if there are lots
of nodes in a DAG).

We could introduce more flexible "binding" mechanism, for example labeling tasks and workers
or using some other "awareness" algorithm.

  was:
Let's assume that we have a cluster with some "specialized" nodes (for
example a worker on IoT device that is responsible for controlling some
actuators, dedicated workers to achieve data locality, some hardware
specific - mem/cpu - tasks, etc.).

There should be a way to "hint" Gearpump to prefer some nodes (workers) over others for certain
tasks (executors). 

We already have "localities" feature. Developer can add a configuration (when submitting 
the application) to bind a task with a worker:
{code}
gearpump {
  streaming {
    localities {
      app1: {            //This is the application name to be submitted
        workerId: [    //The worker id where to launch Task0_0 and Task0_1
          TaskId(0,0), TaskId(0,1)
        ]
      }
    }
  }
}
{code}

The problem is: users have no idea what each Task's task ID is (especially if there are lots
of nodes in a DAG).

We could introduce more flexible "binding" mechanism, for example labeling tasks and workers
or using some other "awareness" algorithm.


> Binding tasks to workers
> ------------------------
>
>                 Key: GEARPUMP-149
>                 URL: https://issues.apache.org/jira/browse/GEARPUMP-149
>             Project: Apache Gearpump
>          Issue Type: Improvement
>            Reporter: Karol Brejna
>            Priority: Minor
>
> Let's assume that we have a cluster with some "specialized" nodes (for example a worker
on IoT device that is responsible for controlling some actuators, dedicated workers to achieve
data locality, some hardware specific - mem/cpu - tasks, etc.).
> There should be a way to "hint" Gearpump to prefer some nodes (workers) over others for
certain tasks (executors). 
> We already have "localities" feature. Developer can add a configuration (when submitting
 the application) to bind a task with a worker:
> {code}
> gearpump {
>   streaming {
>     localities {
>       app1: {            //This is the application name to be submitted
>         workerId: [    //The worker id where to launch Task0_0 and Task0_1
>           TaskId(0,0), TaskId(0,1)
>         ]
>       }
>     }
>   }
> }
> {code}
> The problem is: users have no idea what each Task's task ID is (especially if there are
lots of nodes in a DAG).
> We could introduce more flexible "binding" mechanism, for example labeling tasks and
workers or using some other "awareness" algorithm.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message