Return-Path: X-Original-To: apmail-storm-user-archive@minotaur.apache.org Delivered-To: apmail-storm-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 09BED10A01 for ; Thu, 9 Jan 2014 16:45:03 +0000 (UTC) Received: (qmail 83453 invoked by uid 500); 9 Jan 2014 16:42:14 -0000 Delivered-To: apmail-storm-user-archive@storm.apache.org Received: (qmail 83385 invoked by uid 500); 9 Jan 2014 16:42:03 -0000 Mailing-List: contact user-help@storm.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@storm.incubator.apache.org Delivered-To: mailing list user@storm.incubator.apache.org Received: (qmail 83336 invoked by uid 99); 9 Jan 2014 16:41:58 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jan 2014 16:41:58 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of klaus.schaefers@gmail.com designates 209.85.212.173 as permitted sender) Received: from [209.85.212.173] (HELO mail-wi0-f173.google.com) (209.85.212.173) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jan 2014 16:41:54 +0000 Received: by mail-wi0-f173.google.com with SMTP id hn9so7186877wib.0 for ; Thu, 09 Jan 2014 08:41:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=YuayaRA88c2l1XIT3B8jqXzUodusN9d/1DBjDYRb5jM=; b=KE0knUf9NtzgppN8ibwIPV1Aaoqx3XH7AytpztobliQZ0voldLFV2XXTczYd9NdwPD AkBaDn5Hr9ftQ31Cf1qPkgA28FkTmNdlAqjYiWxJR1QPLx/WMJ2StCsm6pCUk0+nt2uu gNQd+SeYWhiTPCl1KzMaYx9V1rgo1IwBgwLXRyJ3kZytI/gTaply9cRRXjfrTjSJ6uhw w15derpPehAYI68U88XWoUoag0YVvOxk5XEcK2Do2QFroARclooQlYCQSSXJeHBh2Bzy gLvihl6c9zdJl8s1SDGggfwZl56QLi7jJN810ovHvDLqYklQ6z4V9/TvlaFSG9Gh/hZj uFlg== MIME-Version: 1.0 X-Received: by 10.180.184.105 with SMTP id et9mr26602083wic.36.1389285693097; Thu, 09 Jan 2014 08:41:33 -0800 (PST) Received: by 10.180.97.131 with HTTP; Thu, 9 Jan 2014 08:41:33 -0800 (PST) In-Reply-To: References: Date: Thu, 9 Jan 2014 17:41:33 +0100 Message-ID: Subject: Re: Strom research suggestions From: Klausen Schaefersinho To: user@storm.incubator.apache.org Content-Type: multipart/alternative; boundary=001a11c228a23a8ece04ef8c4850 X-Virus-Checked: Checked by ClamAV on apache.org --001a11c228a23a8ece04ef8c4850 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Hi, >* install a distributed state store (e..g cassandra) on the same nodes as the Storm workers > >* try to align the Storm partitioning triggered by the groupby with Cassandra partitioning, so that under usual happy circumstances (no crash), the Storm reduction is happening on the node where Cassandra is storing that particular primary >key, avoiding the network travel for the persistence. I think this would be great feature of storm. Have a reliably and fast way to store state would be great! Cheers, Klaus On Thu, Jan 9, 2014 at 5:11 PM, Adam Lewis wrote: > I love it; even if it is a premature optimization the beauty of academic > work is that this should be measurable and is still an interesting findin= g > either way. I don't have the large scale production experience with stor= m > that others here have (yet), but it sounds like it would really help > performance since you're going after network transfer. And as you say, > Svend, all the ingredients are already built in to trident. > > Adam > > > On Thu, Jan 9, 2014 at 10:56 AM, Brian O'Neill wro= te: > >> >> +1, love the idea. I=92ve wanted to play with partitioning alignment >> myself (with C*), but i=92ve been too busy with the day job. =3D) >> >> Tobias, if you need some support =97 don=92t hesitate to reach out. >> >> If you are able to align the partitioning, and we can add =93in-place=94 >> computation within Storm, it would be great to see a speed comparison >> between Hadoop and Storm. (If comparable, it may drive people to aband= on >> their Hadoop infrastructure for batch processing, and run everything on >> Storm) >> >> -brian >> >> --- >> >> Brian O'Neill >> >> Chief Architect >> >> *Health Market Science* >> >> *The Science of Better Results* >> >> 2700 Horizon Drive =95 King of Prussia, PA =95 19406 >> >> M: 215.588.6024 =95 @boneill42 =95 >> >> healthmarketscience.com >> >> >> This information transmitted in this email message is for the intended >> recipient only and may contain confidential and/or privileged material. = If >> you received this email in error and are not the intended recipient, or = the >> person responsible to deliver it to the intended recipient, please conta= ct >> the sender at the email above and delete this email and any attachments = and >> destroy any copies thereof. Any review, retransmission, dissemination, >> copying or other use of, or taking any action in reliance upon, this >> information by persons or entities other than the intended recipient is >> strictly prohibited. >> >> >> >> >> From: Svend Vanderveken >> Reply-To: >> Date: Thursday, January 9, 2014 at 10:46 AM >> To: >> Subject: Re: Strom research suggestions >> >> Hey Tobias, >> >> >> Nice project, I would have loved to play with something like storm back >> in my university days :) >> >> Here's a topic that's been on my mind for a while (Trident API of storm)= : >> >> >> * one core idea of distributed map reduce =E0 la hadoop was to perform a= s >> much processing as possible close to the data: you execute the "map" >> locally on each node where the data sits, you do a first reduce there, t= hen >> you let the result travel through the network, you do one last reduce >> centrally and you have a result without having all your DB travel the >> network everytime >> >> * Storm groupBy + persistentAggregate + reducer/combiner let us have a >> similar semantic, where we map incoming tuples, reduce them with other >> tuples in the same group + with previously reduced value stored in DB at >> regular interval >> >> * for each group, the operation above happens always on the same Storm >> Task (i.e. the same "place" in the cluster) and stores its ongoing state= in >> the "same place" in DB, using the group value as primary key >> >> I believe it might be worth investigating if the following pattern would >> make sense: >> >> * install a distributed state store (e..g cassandra) on the same nodes a= s >> the Storm workers >> >> * try to align the Storm partitioning triggered by the groupby with >> Cassandra partitioning, so that under usual happy circumstances (no cras= h), >> the Storm reduction is happening on the node where Cassandra is storing >> that particular primary key, avoiding the network travel for the >> persistence. >> >> >> What do you think? Premature optimization? Does not make sense? Great >> idea? Let me know :) >> >> >> S >> >> >> >> >> On Thu, Jan 9, 2014 at 3:00 PM, Tobias Pazer wrot= e: >> >>> Hi all, >>> >>> I have recently started writing my master thesis with a focus on storm, >>> as we are planning to implement the lambda architecture in our universi= ty. >>> >>> As it's still not very clear for me where exactly it's worth to dive >>> into, I was hoping one of you might have any suggestions. >>> >>> I was thinking about a benchmark or something else to systematically >>> evaluate and improve the configuration of storm, but I'm not sure if th= is >>> is even worth the time. >>> >>> I think the more experienced of you definitely have further ideas! >>> >>> Thanks and regards >>> Tobias >>> >> >> > --001a11c228a23a8ece04ef8c4850 Content-Type: text/html; charset=windows-1252 Content-Transfer-Encoding: quoted-printable
Hi,

>* install a distributed state store (e..g cas= sandra) on the same nodes as the Storm workers
>
>= * try to align the Storm partitioning triggered by the groupby with Cassand= ra partitioning, so that under usual happy circumstances (no crash), the St= orm reduction is happening on the node where Cassandra is storing that part= icular primary >key, avoiding the network travel for the persistence.=A0=

I think this= would be great feature of storm. Have a reliably and fast way to store sta= te would be great!

Cheers,

Klaus



On T= hu, Jan 9, 2014 at 5:11 PM, Adam Lewis <mail@adamlewis.com>= wrote:
I love it; even if it is= a premature optimization the beauty of academic work is that this should b= e measurable and is still an interesting finding either way. =A0I don't= have the large scale production experience with storm that others here hav= e (yet), but it sounds like it would really help performance since you'= re going after network transfer. =A0And as you say, Svend, all the ingredie= nts are already built in to trident.

Adam


On Thu, Jan 9, 2014 at 10:56 AM, Brian O'Neill <bone@alumni.brown.= edu> wrote:

+1, love the idea. =A0I=92ve wanted to play= with partitioning alignment myself (with C*), but i=92ve been too busy wit= h the day job. =3D)

Tobias, if you need some support =97 don=92t hesitate t= o reach out.

If you are able to align the partitio= ning, and we can add =93in-place=94 computation within Storm, it would be g= reat to see a speed comparison between Hadoop and Storm. =A0 (If comparable= , it may drive people to abandon their Hadoop infrastructure for batch proc= essing, and run everything on Storm)

-brian

---

= Brian O'Neill

Chief Architect

Health Market=A0Science<= /u>

The Science of Better Results

=

2700 Horizon Drive=A0=95=A0King of Prussia, PA=A0=95=A019406

M: 2= 15.588.6024=A0=95 @boneil= l42=A0=A0=95=A0=A0

healthmarketscience.com=


This information transmitted in= this email message is for the intended recipient only and may contain conf= idential and/or privileged material. If you received this email in error an= d are not the intended recipient, or the person responsible to deliver it t= o the intended recipient, please contact the sender at the email above and = delete this email and any attachments and destroy any copies thereof. Any r= eview, retransmission, dissemination, copying or other use of, or taking an= y action in reliance upon, this information by persons or entities other th= an the intended recipient is strictly prohibited.

=A0


From: Svend Vanderveken <svend.vanderveke= n@gmail.com>
Reply-To: &= lt;use= r@storm.incubator.apache.org>
Date: Thursday, January 9, 2014 at= 10:46 AM
To: <user@storm.incubato= r.apache.org>
Subject: Re: Strom research sugges= tions

Hey Tobias,=A0

Nice project, I would have loved to play wit= h something like storm back in my university days :)

Here's a topic that's been on my mind for a whi= le (Trident API of storm):


* one co= re idea of distributed map reduce =E0 la hadoop was to perform as much proc= essing as possible close to the data: you execute the "map" local= ly on each node where the data sits, you do a first reduce there, then you = let the result travel through the network, you do one last reduce centrally= and you have a result without having all your DB travel the network everyt= ime=A0

* Storm groupBy + persistentAggregate + reducer/combine= r let us have a similar semantic, where we map incoming tuples, reduce them= with other tuples in the same group + with previously reduced value stored= in DB at regular interval=A0

* for each group, the operation above happens always on= the same Storm Task (i.e. the same "place" in the cluster) and s= tores its ongoing state in the "same place" in DB, using the grou= p value as primary key=A0

I believe it might be worth investigating if the follow= ing pattern would make sense:=A0

* install a distr= ibuted state store (e..g cassandra) on the same nodes as the Storm workers<= /div>

* try to align the Storm partitioning triggered by the = groupby with Cassandra partitioning, so that under usual happy circumstance= s (no crash), the Storm reduction is happening on the node where Cassandra = is storing that particular primary key, avoiding the network travel for the= persistence.=A0


What do you think? Premature optimizatio= n? Does not make sense? Great idea? Let me know :)


S




On Thu, Jan 9, 2014 at 3:00 PM, Tobias P= azer <tobiaspazer@gmail.com> wrote:

Hi all,

I have recently started writing my= master thesis with a focus on storm, as we are planning to implement the l= ambda architecture in our university.

As it's still n= ot very clear for me where exactly it's worth to dive into, I was hopin= g one of you might have any suggestions.

I was thinking about a benchmark or something else to system= atically evaluate and improve the configuration of storm, but I'm not s= ure if this is even worth the time.

I think the more expe= rienced of you definitely have further ideas!

Thanks and regards
Tobias




--001a11c228a23a8ece04ef8c4850--