gearpump-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vincent Wang <fvunic...@gmail.com>
Subject Re: Algorithm for assigning Executor to Workers?
Date Wed, 11 May 2016 08:11:18 GMT
You can add a configuration when submitting the application, the format is:

gearpump {
  streaming {
    localities {
      app1: {            //This is the application name to be submitted
        workerId: [    //The worker id where to launch Task0_0 and Task0_1
          TaskId(0,0), TaskId(0,1)
        ]
      }
    }
  }
}

But I think the current way is not flexible when there are lots of nodes in
a DAG because user have no idea what each Task's task ID is.

Thanks,
Huafeng

2016-05-11 15:49 GMT+08:00 Karol Brejna <karolbrejna@apache.org>:

> What are the criteria that decide which Executor/Tasks goes to which
> Workers right now?
>
> I am considering the following scenario.
>
> Let's assume that we have a cluster with some "specialized" nodes (for
> example a worker on IoT device that is responsible for controlling some
> actuators, dedicated workers to achieve data locality, some hardware
> specific - mem/cpu - tasks, etc.).
>
> Are we able to "hint" Gearpump for certain tasks to prefer some nodes over
> others? (For example by labeling tasks and nodes or using some other
> "awareness" mechanism?)
>
> Regards,
> Karol
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message