Return-Path: X-Original-To: apmail-storm-user-archive@minotaur.apache.org Delivered-To: apmail-storm-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 732331017E for ; Wed, 5 Feb 2014 23:42:06 +0000 (UTC) Received: (qmail 38347 invoked by uid 500); 5 Feb 2014 23:42:06 -0000 Delivered-To: apmail-storm-user-archive@storm.apache.org Received: (qmail 38324 invoked by uid 500); 5 Feb 2014 23:42:05 -0000 Mailing-List: contact user-help@storm.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@storm.incubator.apache.org Delivered-To: mailing list user@storm.incubator.apache.org Received: (qmail 38316 invoked by uid 99); 5 Feb 2014 23:42:05 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 Feb 2014 23:42:05 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of uzunbaz@gmail.com designates 209.85.216.171 as permitted sender) Received: from [209.85.216.171] (HELO mail-qc0-f171.google.com) (209.85.216.171) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 Feb 2014 23:42:00 +0000 Received: by mail-qc0-f171.google.com with SMTP id n7so1923111qcx.16 for ; Wed, 05 Feb 2014 15:41:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=hIetlkHJbZDAaKq+4afh1mMCCVsQXwNM5mXTvN4pVd0=; b=pjcSren+wKsPAQinogAnTIjX8HtWVPDvsNKBA6D/qWfGbsb5yIUo5HUEQ71WCPlHga +/b0gxGKfV/DLC8LTaAUO8n9e6MuZN4bUFbb9hiT+LVRfdon5VSN+zoyfWJsyxyqio8n ZzFjauuy8HYFuy7QxWaY770ucV15gpPna/mlTPntuEXiH1QGZEs6mI8PrplhhXO/lFd6 3ZZ4623Odm2aOdCDDyLwWKDIb6DEZiLNXhRxu6B69CI2ZHgDvsSGjuOhv2SKYcL+AGFi 6HP65tatJcKtK60S++7JpDv/p3/jGm98Km46ozYidQjwkTpttedGe0So9bmKL7Dasqau 05Ag== MIME-Version: 1.0 X-Received: by 10.140.100.196 with SMTP id s62mr6970292qge.115.1391643699251; Wed, 05 Feb 2014 15:41:39 -0800 (PST) Received: by 10.224.182.132 with HTTP; Wed, 5 Feb 2014 15:41:39 -0800 (PST) In-Reply-To: References: <20140205160633.GE2505@animetrics.com> Date: Wed, 5 Feb 2014 15:41:39 -0800 Message-ID: Subject: Re: Can a topology be configured to force a maximum of 1 executor per worker? From: Serkan Uzunbaz To: user@storm.incubator.apache.org Content-Type: multipart/alternative; boundary=001a1134f40c59111f04f1b14c2f X-Virus-Checked: Checked by ClamAV on apache.org --001a1134f40c59111f04f1b14c2f Content-Type: text/plain; charset=ISO-8859-1 You can also write your own scheduler and assign your non-thread-safe bolt tasks to different worker processes. Then, use the EvenScheduler to schedule the rest of the tasks. But note that EvenScheduler will not assign any other tasks to those worker processes (the ones that you assigned). There is a tutorial at https://xumingming.sinaapp.com/885/twitter-storm-how-to-develop-a-pluggable-scheduler/ On Wed, Feb 5, 2014 at 8:15 AM, bijoy deb wrote: > Hi Marc, > > I believe keeping the total number of executors(i.e. parallelism) across > all the components(bolts,spouts) to be less than or equal to the total > number of workers can be one way to achieve this. > > Thanks > Bijoy > > > On Wed, Feb 5, 2014 at 9:36 PM, Marc Vaillant wrote: > >> Suppose that you have a bolt whose tasks are not thread safe but you >> still want parallelism. It seems that this could be achieved via >> multiprocessing by forcing a maximium of 1 executor per worker. With >> this constraint, if you chose a parallelism hint of 4 (with default >> executors) you would get 4 tasks in 4 executors each running in a >> separate worker. Can this constraint be configured? >> >> Thanks, >> Marc >> > > --001a1134f40c59111f04f1b14c2f Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
You can also write your own scheduler and assign your= non-thread-safe bolt tasks to different worker processes. Then, use the Ev= enScheduler to schedule the rest of the tasks. But note that EvenScheduler = will not assign any other tasks to those worker processes (the ones that yo= u assigned).
There is a tutorial at https://xumingming.sin= aapp.com/885/twitter-storm-how-to-develop-a-pluggable-scheduler/


On Wed, Feb 5= , 2014 at 8:15 AM, bijoy deb <bijoy.computers@gmail.com> wrote:
Hi Marc,

I believe keeping the total number of executors(i.e. parallelism) across a= ll the components(bolts,spouts) to be less than or equal to the total numbe= r of workers can be one way to achieve this.

Thanks
Bijoy


On Wed, = Feb 5, 2014 at 9:36 PM, Marc Vaillant <vaillant@animetrics.com&g= t; wrote:
Suppose that you have a bolt whose tasks are= not thread safe but you
still want parallelism. =A0It seems that this could be achieved via
multiprocessing by forcing a maximium of 1 executor per worker. =A0With
this constraint, if you chose a parallelism hint of 4 (with default
executors) you would get 4 tasks in 4 executors each running in a
separate worker. =A0Can this constraint be configured?

Thanks,
Marc


--001a1134f40c59111f04f1b14c2f--