Mailing-List: contact user-help@storm.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@storm.incubator.apache.org
Received-SPF: pass (athena.apache.org: domain of svend.vanderveken@gmail.com
 designates 209.85.216.181 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CANW+zHpUDmR9p6RshRfbg0k3coiozPLJtRsbNJ21yzGTANR2Vg@mail.gmail.com>
References: 
 <CANW+zHrL3hspzNux=L-AUSUq++nMLJ99tjtGTYNCAnQJiT+9Ww@mail.gmail.com>
	<CANW+zHpUDmR9p6RshRfbg0k3coiozPLJtRsbNJ21yzGTANR2Vg@mail.gmail.com>
Date: Thu, 9 Jan 2014 16:46:31 +0100
Message-ID: 
 <CAAYLz+rz3rNp+VObu7KzvW97uR840asX-cczfQZZJS0KfPxp7w@mail.gmail.com>
Subject: Re: Strom research suggestions
From: Svend Vanderveken <svend.vanderveken@gmail.com>
To: user@storm.incubator.apache.org
Content-Type: multipart/alternative; boundary=047d7bd76ef26eb22004ef8b83b5

--047d7bd76ef26eb22004ef8b83b5
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hey Tobias,


Nice project, I would have loved to play with something like storm back in
my university days :)

Here's a topic that's been on my mind for a while (Trident API of storm):


* one core idea of distributed map reduce =E0 la hadoop was to perform as
much processing as possible close to the data: you execute the "map"
locally on each node where the data sits, you do a first reduce there, then
you let the result travel through the network, you do one last reduce
centrally and you have a result without having all your DB travel the
network everytime

* Storm groupBy + persistentAggregate + reducer/combiner let us have a
similar semantic, where we map incoming tuples, reduce them with other
tuples in the same group + with previously reduced value stored in DB at
regular interval

* for each group, the operation above happens always on the same Storm Task
(i.e. the same "place" in the cluster) and stores its ongoing state in the
"same place" in DB, using the group value as primary key

I believe it might be worth investigating if the following pattern would
make sense:

* install a distributed state store (e..g cassandra) on the same nodes as
the Storm workers

* try to align the Storm partitioning triggered by the groupby with
Cassandra partitioning, so that under usual happy circumstances (no crash),
the Storm reduction is happening on the node where Cassandra is storing
that particular primary key, avoiding the network travel for the
persistence.


What do you think? Premature optimization? Does not make sense? Great idea?
Let me know :)


S


On Thu, Jan 9, 2014 at 3:00 PM, Tobias Pazer <tobiaspazer@gmail.com> wrote:

> Hi all,
>
> I have recently started writing my master thesis with a focus on storm, a=
s
> we are planning to implement the lambda architecture in our university.
>
> As it's still not very clear for me where exactly it's worth to dive into=
,
> I was hoping one of you might have any suggestions.
>
> I was thinking about a benchmark or something else to systematically
> evaluate and improve the configuration of storm, but I'm not sure if this
> is even worth the time.
>
> I think the more experienced of you definitely have further ideas!
>
> Thanks and regards
> Tobias
>

--047d7bd76ef26eb22004ef8b83b5
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hey Tobias,=A0<div><br></div><div><br></div><div>Nice proj=
ect, I would have loved to play with something like storm back in my univer=
sity days :)</div><div><br></div><div>Here&#39;s a topic that&#39;s been on=
 my mind for a while (Trident API of storm):</div>
<div><br></div><div><br></div><div>* one core idea of distributed map reduc=
e =E0 la hadoop was to perform as much processing as possible close to the =
data: you execute the &quot;map&quot; locally on each node where the data s=
its, you do a first reduce there, then you let the result travel through th=
e network, you do one last reduce centrally and you have a result without h=
aving all your DB travel the network everytime=A0</div>
<div><br></div><div>* Storm groupBy + persistentAggregate + reducer/combine=
r let us have a similar semantic, where we map incoming tuples, reduce them=
 with other tuples in the same group + with previously reduced value stored=
 in DB at regular interval=A0</div>
<div><br></div><div>* for each group, the operation above happens always on=
 the same Storm Task (i.e. the same &quot;place&quot; in the cluster) and s=
tores its ongoing state in the &quot;same place&quot; in DB, using the grou=
p value as primary key=A0</div>
<div><br></div><div>I believe it might be worth investigating if the follow=
ing pattern would make sense:=A0</div><div><br></div><div>* install a distr=
ibuted state store (e..g cassandra) on the same nodes as the Storm workers<=
/div>
<div><br></div><div>* try to align the Storm partitioning triggered by the =
groupby with Cassandra partitioning, so that under usual happy circumstance=
s (no crash), the Storm reduction is happening on the node where Cassandra =
is storing that particular primary key, avoiding the network travel for the=
 persistence.=A0</div>
<div><br></div><div><br></div><div>What do you think? Premature optimizatio=
n? Does not make sense? Great idea? Let me know :)</div><div><br></div><div=
><br></div><div>S</div><div><br></div><div><br></div></div><div class=3D"gm=
ail_extra">
<br><br><div class=3D"gmail_quote">On Thu, Jan 9, 2014 at 3:00 PM, Tobias P=
azer <span dir=3D"ltr">&lt;<a href=3D"mailto:tobiaspazer@gmail.com" target=
=3D"_blank">tobiaspazer@gmail.com</a>&gt;</span> wrote:<br><blockquote clas=
s=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;pad=
ding-left:1ex">
<p dir=3D"ltr">Hi all,</p>
<p dir=3D"ltr">I have recently started writing my master thesis with a focu=
s on storm, as we are planning to implement the lambda architecture in our =
university.</p>
<p dir=3D"ltr">As it&#39;s still not very clear for me where exactly it&#39=
;s worth to dive into, I was hoping one of you might have any suggestions. =
</p>
<p dir=3D"ltr">I was thinking about a benchmark or something else to system=
atically evaluate and improve the configuration of storm, but I&#39;m not s=
ure if this is even worth the time.</p>
<p dir=3D"ltr">I think the more experienced of you definitely have further =
ideas!</p>
<p dir=3D"ltr">Thanks and regards <br><span class=3D"HOEnZb"><font color=3D=
"#888888">
Tobias</font></span></p>
</blockquote></div><br></div>

--047d7bd76ef26eb22004ef8b83b5--