Return-Path: X-Original-To: apmail-storm-user-archive@minotaur.apache.org Delivered-To: apmail-storm-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 61E3F11758 for ; Wed, 16 Jul 2014 23:41:46 +0000 (UTC) Received: (qmail 86308 invoked by uid 500); 16 Jul 2014 23:41:45 -0000 Delivered-To: apmail-storm-user-archive@storm.apache.org Received: (qmail 86268 invoked by uid 500); 16 Jul 2014 23:41:45 -0000 Mailing-List: contact user-help@storm.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@storm.incubator.apache.org Delivered-To: mailing list user@storm.incubator.apache.org Received: (qmail 86258 invoked by uid 99); 16 Jul 2014 23:41:45 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Jul 2014 23:41:45 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of andreas.grammenos@gmail.com designates 209.85.216.54 as permitted sender) Received: from [209.85.216.54] (HELO mail-qa0-f54.google.com) (209.85.216.54) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Jul 2014 23:41:42 +0000 Received: by mail-qa0-f54.google.com with SMTP id k15so1272169qaq.41 for ; Wed, 16 Jul 2014 16:41:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type; bh=W+MOzriKcuk3A3YdhlqrvfyyK0Ak5GvajWuxM5LNZvE=; b=0xeTS09alJ1Iu+QfJRoGNTNgxLFhw5TBgKyjjr270AYoBPpED0gffYyODn8NsJBs05 tylmll2Ei5jNI9vfPYf3GJ2lgSMhf8/9EyNnHDUNSsU9t34j7fSW5FvQyuOyKgKcSFN4 rjoFP5N6Pagd2HUOIxw0uCEkD0hllRGm9Xvu1tauhRHrm+H7FwzwkTbabZwbw0arDw1Z y71lllyA+YQXnXmp00KJzg49hOyR4uHEOOB3RMBpt7QnDUJOgEK5g0JK39tORZ8ft7jo Os7WV/LdNVC3dkQg4dC5yl3jww2mvHFO42lqt5IoGnVbH9wBITECzyoJdQw7u/n0EsOM HUrg== X-Received: by 10.229.140.70 with SMTP id h6mr2416107qcu.3.1405554077834; Wed, 16 Jul 2014 16:41:17 -0700 (PDT) MIME-Version: 1.0 Received: by 10.96.75.69 with HTTP; Wed, 16 Jul 2014 16:40:57 -0700 (PDT) In-Reply-To: References: From: Andrew Xor Date: Thu, 17 Jul 2014 02:40:57 +0300 Message-ID: Subject: Re: Distribute Spout output among all bolts To: user@storm.incubator.apache.org Content-Type: multipart/alternative; boundary=001a1132ec3285a53f04fe580f5b X-Virus-Checked: Checked by ClamAV on apache.org --001a1132ec3285a53f04fe580f5b Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable =E2=80=8B Hi Tomas, As I said in my previous mail the grouping is for a bolt *task* not for the actual number of spawned bolts; for example let's say you have two bolts that have a parallelism hint of 3 and these two bolts are wired to the same spout. If you set the bolts as such: tb.setBolt("b1", new ExampleBolt(), 2 /* p-hint */).shuffleGrouping("spout1"); tb.setBolt("b2", new ExampleBolt(), 2 /* p-hint */).shuffleGrouping("spout1"); Then each of the tasks will receive half of the spout tuples but each actual spawned bolt will receive all of the tuples emitted from the spout. This is more evident if you set up a counter in the bolt counting how many tuples if has received and testing this with no parallelism hint as such: tb.setBolt("b1", new ExampleBolt(),).shuffleGrouping("spout1"); tb.setBolt("b2", new ExampleBolt()).shuffleGrouping("spout1"); Now you will see that both bolts will receive all tuples emitted by spout1. Hope this helps. =E2=80=8B =E2=80=8BAndrew.=E2=80=8B On Thu, Jul 17, 2014 at 2:33 AM, Tomas Mazukna wrote: > Andrew, > > when you connect your bolt to your spout you specify the grouping. If you > use shuffle grouping then any free bolt gets the tuple - in my experience > even in lightly loaded topologies the distribution amongst bolts is prett= y > even. If you use all grouping then all bolts receive a copy of the tuple. > Use shuffle grouping and each of your bolts will get about 1/3 of the > workload. > > Tomas > > > On Wed, Jul 16, 2014 at 7:05 PM, Andrew Xor > wrote: > >> H >> =E2=80=8Bi, >> >> I am trying to distribute the spout output to it's subscribed bolts >> evenly; let's say that I have a spout that emits tuples and three bolts >> that are subscribed to it. I want each of the three bolts to receive 1/3 >> rth of the output (or emit a tuple to each one of these bolts in turns). >> Unfortunately as far as I understand all bolts will receive all of the >> emitted tuples of that particular spout regardless of the grouping defin= ed >> (as grouping from my understanding is for bolt *tasks* not actual bolts)= . >> >> I've searched a bit and I can't seem to find a way to accomplish >> that...=E2=80=8B is there a way to do that or I am searching in vain? >> >> Thanks. >> > > > > -- > Tomas Mazukna > 678-557-3834 > --001a1132ec3285a53f04fe580f5b Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
=E2=80= =8B
Hi Tomas,

=C2=A0As I said in my previous mail the grouping is for a bolt *task* not for=20 the actual number of spawned bolts; for example let's say you have two = bolts that=20 have a parallelism hint of 3 and these two bolts are wired to the same=20 spout. If you set the bolts as such:

tb.setBolt("b1", new = ExampleBolt(), 2 /* p-hint */).shuffleGrouping("spout1");
tb.s= etBolt("b2", new ExampleBolt(), 2 /* p-hint */).shuffleGrouping(&= quot;spout1");

Then each of the tasks will receive half of the spout tuples but each actual spawned bolt will receive all of the tuples emitted from the spout.=20 This is more evident if you set up a counter in the bolt counting how=20 many tuples if has received and testing this with no parallelism hint as such:

tb.setBolt("b1", new ExampleBolt(),).shuffleGroupin= g("spout1");
tb.setBolt("b2", new ExampleBolt()).shu= ffleGrouping("spout1");

Now you will see that both bolts will receive all tuples emitted by spout1.=

Hope this helps.

=E2=80=8B
=E2=80=8BAndrew.=E2=80=8B


On Thu, Jul 17, 2014 at 2:33 AM, Tomas Mazukna <t= omas.mazukna@gmail.com> wrote:
Andrew,<= div>
when you connect your bolt to your spout you specify the= grouping. If you use shuffle grouping then any free bolt gets the tuple - = in my experience even in lightly loaded topologies the distribution amongst= bolts is pretty even. If you use all grouping then all bolts receive a cop= y of the tuple.=C2=A0
Use shuffle grouping and each of your bolts will get about 1/3 of the = workload.

Tomas


On Wed, Jul 1= 6, 2014 at 7:05 PM, Andrew Xor <andreas.grammenos@gmail.com&= gt; wrote:
H
=E2=80=8Bi,

=C2=A0I am trying to distribute the spout output to it's subscribed bol= ts evenly; let's say that I have a spout that emits tuples and three bo= lts that are subscribed to it. I want each of the three bolts to receive 1/= 3 rth of the output (or emit a tuple to each one of these bolts in turns). = Unfortunately as far as I understand all bolts will receive all of the emit= ted tuples of that particular spout regardless of the grouping defined (as = grouping from my understanding is for bolt *tasks* not actual bolts).

=C2=A0I've searched a bit and I ca= n't seem to find a way to accomplish that...=E2=80=8B is there a way to= do that or I am searching in vain?

Thanks.



--
Tomas Mazukna
678-557-3834

--001a1132ec3285a53f04fe580f5b--