storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Serkan Uzunbaz <uzun...@gmail.com>
Subject Re: Can a topology be configured to force a maximum of 1 executor per worker?
Date Wed, 05 Feb 2014 23:41:39 GMT
You can also write your own scheduler and assign your non-thread-safe bolt
tasks to different worker processes. Then, use the EvenScheduler to
schedule the rest of the tasks. But note that EvenScheduler will not assign
any other tasks to those worker processes (the ones that you assigned).
There is a tutorial at
https://xumingming.sinaapp.com/885/twitter-storm-how-to-develop-a-pluggable-scheduler/


On Wed, Feb 5, 2014 at 8:15 AM, bijoy deb <bijoy.computers@gmail.com> wrote:

> Hi Marc,
>
> I believe keeping the total number of executors(i.e. parallelism) across
> all the components(bolts,spouts) to be less than or equal to the total
> number of workers can be one way to achieve this.
>
> Thanks
> Bijoy
>
>
> On Wed, Feb 5, 2014 at 9:36 PM, Marc Vaillant <vaillant@animetrics.com>wrote:
>
>> Suppose that you have a bolt whose tasks are not thread safe but you
>> still want parallelism.  It seems that this could be achieved via
>> multiprocessing by forcing a maximium of 1 executor per worker.  With
>> this constraint, if you chose a parallelism hint of 4 (with default
>> executors) you would get 4 tasks in 4 executors each running in a
>> separate worker.  Can this constraint be configured?
>>
>> Thanks,
>> Marc
>>
>
>

Mime
View raw message