gearpump-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Manu Zhang <owenzhang1...@gmail.com>
Subject Re: Algorithm for assigning Executor to Workers?
Date Wed, 11 May 2016 08:19:08 GMT
Hi Karol,

Weihua has previously written up this nice doc IoT features map with
Gearpump
<https://docs.google.com/document/d/1kagryYsBmtwVIE5E8afgH9K_O2a0_VdtlAY9auVAQQM/edit?usp=sharing>.
You may find more details about how Gearpump can work in the environment of
cluster and edge devices and the gaps to fill.

On Wed, May 11, 2016 at 4:11 PM Vincent Wang <fvunicorn@gmail.com> wrote:

> You can add a configuration when submitting the application, the format is:
>
> gearpump {
>   streaming {
>     localities {
>       app1: {            //This is the application name to be submitted
>         workerId: [    //The worker id where to launch Task0_0 and Task0_1
>           TaskId(0,0), TaskId(0,1)
>         ]
>       }
>     }
>   }
> }
>
> But I think the current way is not flexible when there are lots of nodes in
> a DAG because user have no idea what each Task's task ID is.
>
> Thanks,
> Huafeng
>
> 2016-05-11 15:49 GMT+08:00 Karol Brejna <karolbrejna@apache.org>:
>
> > What are the criteria that decide which Executor/Tasks goes to which
> > Workers right now?
> >
> > I am considering the following scenario.
> >
> > Let's assume that we have a cluster with some "specialized" nodes (for
> > example a worker on IoT device that is responsible for controlling some
> > actuators, dedicated workers to achieve data locality, some hardware
> > specific - mem/cpu - tasks, etc.).
> >
> > Are we able to "hint" Gearpump for certain tasks to prefer some nodes
> over
> > others? (For example by labeling tasks and nodes or using some other
> > "awareness" mechanism?)
> >
> > Regards,
> > Karol
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message