storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marc Vaillant <vaill...@animetrics.com>
Subject Re: Can a topology be configured to force a maximum of 1 executor per worker?
Date Thu, 06 Feb 2014 17:58:27 GMT
Great, thanks!

On Wed, Feb 05, 2014 at 03:41:39PM -0800, Serkan Uzunbaz wrote:
> You can also write your own scheduler and assign your non-thread-safe bolt
> tasks to different worker processes. Then, use the EvenScheduler to schedule
> the rest of the tasks. But note that EvenScheduler will not assign any other
> tasks to those worker processes (the ones that you assigned).
> There is a tutorial at https://xumingming.sinaapp.com/885/
> twitter-storm-how-to-develop-a-pluggable-scheduler/
> 
> 
> On Wed, Feb 5, 2014 at 8:15 AM, bijoy deb <bijoy.computers@gmail.com> wrote:
> 
>     Hi Marc,
> 
>     I believe keeping the total number of executors(i.e. parallelism) across
>     all the components(bolts,spouts) to be less than or equal to the total
>     number of workers can be one way to achieve this.
> 
>     Thanks
>     Bijoy
>    
> 
>     On Wed, Feb 5, 2014 at 9:36 PM, Marc Vaillant <vaillant@animetrics.com>
>     wrote:
> 
>         Suppose that you have a bolt whose tasks are not thread safe but you
>         still want parallelism.  It seems that this could be achieved via
>         multiprocessing by forcing a maximium of 1 executor per worker.  With
>         this constraint, if you chose a parallelism hint of 4 (with default
>         executors) you would get 4 tasks in 4 executors each running in a
>         separate worker.  Can this constraint be configured?
> 
>         Thanks,
>         Marc
> 
> 
> 
> 

Mime
View raw message