Return-Path: X-Original-To: apmail-storm-user-archive@minotaur.apache.org Delivered-To: apmail-storm-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id CDB07109F1 for ; Wed, 15 Jan 2014 18:08:05 +0000 (UTC) Received: (qmail 60768 invoked by uid 500); 15 Jan 2014 18:08:05 -0000 Delivered-To: apmail-storm-user-archive@storm.apache.org Received: (qmail 60697 invoked by uid 500); 15 Jan 2014 18:08:04 -0000 Mailing-List: contact user-help@storm.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@storm.incubator.apache.org Delivered-To: mailing list user@storm.incubator.apache.org Received: (qmail 60689 invoked by uid 99); 15 Jan 2014 18:08:04 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jan 2014 18:08:04 +0000 X-ASF-Spam-Status: No, hits=1.8 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,HTML_MESSAGE,MIME_QP_LONG_LINE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of boneill42@gmail.com designates 209.85.128.46 as permitted sender) Received: from [209.85.128.46] (HELO mail-qe0-f46.google.com) (209.85.128.46) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jan 2014 18:08:00 +0000 Received: by mail-qe0-f46.google.com with SMTP id 8so1434878qea.33 for ; Wed, 15 Jan 2014 10:07:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:user-agent:date:subject:from:to:message-id:thread-topic :references:in-reply-to:mime-version:content-type; bh=wHaeM3avRVK/NZSxbSknynkYPOGImzkNlzCxVzVWrrs=; b=GZDqt1ZLWl7TuNDQRr8op81DLfgP4fE+GFX4oHktmp65DWDR7pBcC9OxUkS0zzNald lH+HrETDfGtHfWN0A25qNpK8vZILDpj2HYFMDB7o2FFbOH5sQbtW69hbM7ctQDHguzQ0 QNKGKWtDp1gfNszZE2iAGH+Y8glDQHp3IvyCPFxn7HcHO9IHobQu/wUJaX/+3T/m+kb8 Eb4tBhWE3rcIQt0NnPqsBdqNnGPXEdMf6ja3gjhQwagOcpae61qMFnw1GCs5gFoXUOLq uztuDAL6r4bZPOcIS/goRrpmpzLQ+f+iwYGn3uT1Zlu0m8C6GsyMdDZEm0BdkNM0Dw8l Zmlg== X-Received: by 10.140.90.80 with SMTP id w74mr471047qgd.96.1389809259481; Wed, 15 Jan 2014 10:07:39 -0800 (PST) Received: from [10.60.71.81] ([67.132.206.254]) by mx.google.com with ESMTPSA id r9sm7777030qey.19.2014.01.15.10.07.36 for (version=TLSv1 cipher=RC4-SHA bits=128/128); Wed, 15 Jan 2014 10:07:38 -0800 (PST) Sender: "Brian O'Neill" User-Agent: Microsoft-MacOutlook/14.3.9.131030 Date: Wed, 15 Jan 2014 13:07:33 -0500 Subject: Re: Strom research suggestions From: Brian O'Neill To: Message-ID: Thread-Topic: Strom research suggestions References: In-Reply-To: Mime-version: 1.0 Content-type: multipart/alternative; boundary="B_3472636057_2920739" X-Virus-Checked: Checked by ClamAV on apache.org > This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. --B_3472636057_2920739 Content-type: text/plain; charset="ISO-8859-1" Content-transfer-encoding: quoted-printable Agree w/ Svend. That use case is a good one where Cassandra is used for output. I=B9d also suggest that you tackle a use case that uses a ColumnFamily as input. =20 (perhaps a Kafka queue of row/partition keys) Then, use Svend=B9s suggestion to route the keys to the machines that host th= e data. -brian --- Brian O'Neill Chief Architect Health Market Science The Science of Better Results 2700 Horizon Drive =80 King of Prussia, PA =80 19406 M: 215.588.6024 =80 @boneill42 =80 healthmarketscience.com This information transmitted in this email message is for the intended recipient only and may contain confidential and/or privileged material. If you received this email in error and are not the intended recipient, or the person responsible to deliver it to the intended recipient, please contact the sender at the email above and delete this email and any attachments and destroy any copies thereof. Any review, retransmission, dissemination, copying or other use of, or taking any action in reliance upon, this information by persons or entities other than the intended recipient is strictly prohibited. =20 From: Svend Vanderveken Reply-To: Date: Friday, January 10, 2014 at 6:39 AM To: Subject: Re: Strom research suggestions Hi,=20 Cool, I hope this thing can get started then. Based the comments from Brian= , Adam, Klausen and Michael 's that I was happy to read, I feel I would not b= e the only one willing to share ideas and/or code about that :D I guess a starting point would be to dig into the details of the available strategies for partitioning data in Cassandra: http://www.datastax.com/docs/1.0/cluster_architecture/partitioning Then imagine you have a bunch of Storm tuples coming in in real time, including, say, a geo-localization, and you want to regroup all the events happening in the same "locationId" (e.g postal code, or rounded latitude/longitude, whatever...) in order to have some counters for each such group. Storm is going to partition the processing of all those tuples across its cluster, so the idea is to tell Storm to do so in the same fashion as Cassandra is partitioning the storage (the counters for each locationId). Hmmm, maybe it's as simple as adding a tuple field that contains the result of the Cassandra partitioner (by querying it or including the logic as a functionality of the topology) and do a partitionBy() on that. Actually, as I understand it groupBy()+PersistentAggregate is built based on a simple partitionBy+partitionAggregate, so code-wise this whole thing might not be huge. Go ahead, that sounds cool! Cheers,=20 Svend On Thu, Jan 9, 2014 at 9:53 PM, Tobias Pazer wrote: > This is exactly what I was looking for, as I am reading a lot about Hadoo= p at > the same time. Haven't got any experience with partitioning alignment so = far, > so I would appreciate any suggestions on how to approach this topic > efficiently. But this shouldn't be a problem as I still have until Octobe= r... >=20 > Now I just have to convince my academic advisor. >=20 > Thanks so far I think this topic is definitly worth to look into. >=20 >=20 >=20 >=20 > 2014/1/9 Michael Oczkowski >> +1 for this idea. I heard DataStax was investigating Storm integration = (like >> they do with Hadoop) but so far as I know this isn=B9t going to happen. T= he >> need for push-down analytics is great and a very general problem and any= nice >> solution would help many people! >> =20 >> Also to Brian=B9s point it would be great to use Storm in lieu of Hadoop i= f >> it=B9s performant. >> =20 >> From: supercargo@gmail.com [mailto:supercargo@gmail.com] On Behalf Of Ad= am >> Lewis >> Sent: Thursday, January 9, 2014 9:11 AM >> To: user >>=20 >>=20 >> Subject: Re: Strom research suggestions >> =20 >>=20 >> I love it; even if it is a premature optimization the beauty of academic= work >> is that this should be measurable and is still an interesting finding ei= ther >> way. I don't have the large scale production experience with storm that >> others here have (yet), but it sounds like it would really help performa= nce >> since you're going after network transfer. And as you say, Svend, all t= he >> ingredients are already built in to trident. >>=20 >> =20 >>=20 >> Adam >>=20 >> =20 >>=20 >> On Thu, Jan 9, 2014 at 10:56 AM, Brian O'Neill w= rote: >>>=20 >>> =20 >>>=20 >>> +1, love the idea. I=B9ve wanted to play with partitioning alignment mys= elf >>> (with C*), but i=B9ve been too busy with the day job. =3D) >>>=20 >>> =20 >>>=20 >>> Tobias, if you need some support =8B don=B9t hesitate to reach out. >>>=20 >>> =20 >>>=20 >>> If you are able to align the partitioning, and we can add =B3in-place=B2 >>> computation within Storm, it would be great to see a speed comparison >>> between Hadoop and Storm. (If comparable, it may drive people to aban= don >>> their Hadoop infrastructure for batch processing, and run everything on >>> Storm) >>>=20 >>> =20 >>>=20 >>> -brian >>>=20 >>> =20 >>>=20 >>> --- >>> Brian O'Neill >>> Chief Architect >>> Health Market Science >>> The Science of Better Results >>> 2700 Horizon Drive =80 King of Prussia, PA =80 19406 >>> M: 215.588.6024 =80 @boneill42 >>> =80 >>> healthmarketscience.com >>> =20 >>> This information transmitted in this email message is for the intended >>> recipient only and may contain confidential and/or privileged material.= If >>> you received this email in error and are not the intended recipient, or= the >>> person responsible to deliver it to the intended recipient, please cont= act >>> the sender at the email above and delete this email and any attachments= and >>> destroy any copies thereof. Any review, retransmission, dissemination, >>> copying or other use of, or taking any action in reliance upon, this >>> information by persons or entities other than the intended recipient is >>> strictly prohibited. >>> =20 >>>=20 >>> =20 >>>=20 >>> From: Svend Vanderveken >> > >>> Reply-To: >> > >>> Date: Thursday, January 9, 2014 at 10:46 AM >>> To: >> > >>> Subject: Re: Strom research suggestions >>>=20 >>> =20 >>>=20 >>> Hey Tobias,=20 >>>=20 >>> =20 >>>=20 >>> =20 >>>=20 >>> Nice project, I would have loved to play with something like storm back= in >>> my university days :) >>>=20 >>> =20 >>>=20 >>> Here's a topic that's been on my mind for a while (Trident API of storm= ): >>>=20 >>> =20 >>>=20 >>> =20 >>>=20 >>> * one core idea of distributed map reduce =E0 la hadoop was to perform as= much >>> processing as possible close to the data: you execute the "map" locally= on >>> each node where the data sits, you do a first reduce there, then you le= t the >>> result travel through the network, you do one last reduce centrally and= you >>> have a result without having all your DB travel the network everytime >>>=20 >>> =20 >>>=20 >>> * Storm groupBy + persistentAggregate + reducer/combiner let us have a >>> similar semantic, where we map incoming tuples, reduce them with other >>> tuples in the same group + with previously reduced value stored in DB a= t >>> regular interval >>>=20 >>> =20 >>>=20 >>> * for each group, the operation above happens always on the same Storm = Task >>> (i.e. the same "place" in the cluster) and stores its ongoing state in = the >>> "same place" in DB, using the group value as primary key >>>=20 >>> =20 >>>=20 >>> I believe it might be worth investigating if the following pattern woul= d >>> make sense:=20 >>>=20 >>> =20 >>>=20 >>> * install a distributed state store (e..g cassandra) on the same nodes = as >>> the Storm workers >>>=20 >>> =20 >>>=20 >>> * try to align the Storm partitioning triggered by the groupby with >>> Cassandra partitioning, so that under usual happy circumstances (no cra= sh), >>> the Storm reduction is happening on the node where Cassandra is storing= that >>> particular primary key, avoiding the network travel for the persistence= . >>>=20 >>> =20 >>>=20 >>> =20 >>>=20 >>> What do you think? Premature optimization? Does not make sense? Great i= dea? >>> Let me know :) >>>=20 >>> =20 >>>=20 >>> =20 >>>=20 >>> S >>>=20 >>> =20 >>>=20 >>> =20 >>>=20 >>> =20 >>>=20 >>> On Thu, Jan 9, 2014 at 3:00 PM, Tobias Pazer >> > wrote: >>>> Hi all, >>>>=20 >>>> I have recently started writing my master thesis with a focus on storm= , as >>>> we are planning to implement the lambda architecture in our university= . >>>>=20 >>>> As it's still not very clear for me where exactly it's worth to dive i= nto, >>>> I was hoping one of you might have any suggestions. >>>>=20 >>>> I was thinking about a benchmark or something else to systematically >>>> evaluate and improve the configuration of storm, but I'm not sure if t= his >>>> is even worth the time. >>>>=20 >>>> I think the more experienced of you definitely have further ideas! >>>>=20 >>>> Thanks and regards >>>> Tobias >>>=20 >>> =20 >> =20 >=20 --B_3472636057_2920739 Content-type: text/html; charset="ISO-8859-1" Content-transfer-encoding: quoted-printable

Agre= e w/ Svend.  That use case is a good one where Cassandra is used for ou= tput.

I’d also suggest that you tackle a use = case that uses a ColumnFamily as input.  
(perhaps a Kafka qu= eue of row/partition keys)

Then, use Svend’s = suggestion to route the keys to the machines that host the data.
<= br>
-brian

= ---

Brian O'Nei= ll

Chief Architect

Health Market Science

= The Science of Better Results

2700 Horizon Drive • King of Prussia, PA <= span style=3D"color: rgb(237, 26, 52); position: relative; top: -0.5pt; letter= -spacing: -0.4pt; ">•&nb= sp;19406

M: 215.588.6= 024 = 226; @boneill42&nb= sp; •  

healthma= rketscience.com


 

<= /div>


Hi, 

C= ool, I hope this thing can get started then. Based the comments from Brian, = Adam, Klausen and Michael 's that I was happy to read, I feel I would not be= the only one willing to share ideas and/or code about that :D
I guess a starting point would be to dig into the details of the= available strategies for partitioning data in Cassandra: 


Then imagine you have a= bunch of Storm tuples coming in in real time, including, say, a geo-localiz= ation, and you want to regroup all the events happening in the same "locatio= nId" (e.g postal code, or rounded latitude/longitude, whatever...) in order = to have some counters for each such group. Storm is going to partition the p= rocessing of all those tuples across its cluster, so the idea is to tell Sto= rm to do so in the same fashion as Cassandra is partitioning the storage (th= e counters for each locationId). Hmmm, maybe it's as simple as adding a tupl= e field that contains the result of the Cassandra partitioner (by querying i= t or including the logic as a functionality of the topology) and do a partit= ionBy() on that. Actually, as I understand it groupBy()+PersistentAggregate = is built based on a simple partitionBy+partitionAggregate, so code-wise this= whole thing might not be huge. Go ahead, that sounds cool!

Cheers, 

Svend





On Thu, Jan 9, 2014 at 9:53 PM, Tobias Pazer = <t= obiaspazer@gmail.com> wrote:
This is exactly what I was looking for, as I am reading a lot abo= ut Hadoop at the same time. Haven't got any experience with partitioning ali= gnment so far, so I would appreciate any suggestions on how to approach this= topic efficiently. But this shouldn't be a problem as I still have until Oc= tober...

Now I just have to convince my academic advisor.=

Thanks so far I think this topic is definitly wort= h to look into.




2014/1/9 Michael Oczkowski <Michael.Oczkowski@seeq.com><= br>

+1 for this idea.  I heard DataSt= ax was investigating Storm integration (like they do with Hadoop) but so far= as I know this isn’t going to happen.  The need for push-down analytics is great and a very general problem and any nice s= olution would help many people!

 

Also to Brian’s point it would be great to use Storm in lieu of H= adoop if it’s performant.

 

From:<= span style=3D"font-size: 11pt; font-family: Calibri, sans-serif;"> supercargo@gmail.com [mailto:= supercargo@gmail.com] On Behalf Of Adam Lewis
Sent: Thursday, January 9, 2014 9:= 11 AM
To: user

 

 

On Thu, Ja= n 9, 2014 at 10:56 AM, Brian O'Neill <bone@alumni.brown.edu> wrote:

 

+1, love the idea.  I’ve wanted to play with partitioning = alignment myself (with C*), but i’ve been too busy with the day job. =3D= )

 

Tobias, if you need some support — don’t he= sitate to reach out.

&nbs= p;

If you are able to align the partiti= oning, and we can add “in-place” computation within Storm, it wo= uld be great to see a speed comparison between Hadoop and Storm.   (If comparable, it may drive people to abandon their Hadoop infrastructure for= batch processing, and run everything on Storm)

 

-brian=

 

---

Brian O'Neill

Chief Architect

Health Market Science

The Science of Better Results<= u>

2700 Horizon Drive • King of Prussia, PA • = 19406

M: 215.588.6024 • @boneill42  •  

healthmarke= tscience.com

 

This in= formation transmitted in this email message is for the intended recipient on= ly and may contain confidential and/or privileged material. If you received this email in error and are not the intended recipient, or the person resp= onsible to deliver it to the intended recipient, please contact the sender a= t the email above and delete this email and any attachments and destroy any = copies thereof. Any review, retransmission, dissemination, copying or other use of, or taking any action in reliance u= pon, this information by persons or entities other than the intended recipie= nt is strictly prohibited.

 

 

<= div style=3D"border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0in 0in = 0in">

From: Svend Vanderveken <svend.vanderveken@gmail.com>
Reply-To: <user@storm.incubator.apache.org= = >
Date: Thursday, January 9, 2014 at 10:46 AM
To: <= ;user@storm.inc= ubator.apache.org>
Subject: Re: Strom research suggestions

 

Hey Tobias, 

 

 =

Nice project, I would have loved to play with= something like storm back in my university days :)

=

 

Here'= s a topic that's been on my mind for a while (Trident API of storm):<= u>

 

 

<= span style=3D"font-size: 9pt; font-family: Verdana, sans-serif;">* one core id= ea of distributed map reduce =E0 la hadoop was to perform as much processing a= s possible close to the data: you execute the "map" locally on each node whe= re the data sits, you do a first reduce there, then you let the result travel thr= ough the network, you do one last reduce centrally and you have a result wit= hout having all your DB travel the network everytime 

 

 

=

* for each group, the operation above happens always on the same St= orm Task (i.e. the same "place" in the cluster) and stores its ongoing state= in the "same place" in DB, using the group value as primary key 

<= div>

 

= I believe it= might be worth investigating if the following pattern would make sense:&nbs= p;

 

* install a distributed state store (e..g cassandra) o= n the same nodes as the Storm workers

 

* try to align the = Storm partitioning triggered by the groupby with Cassandra partitioning, so = that under usual happy circumstances (no crash), the Storm reduction is happ= ening on the node where Cassandra is storing that particular primary key, avoidi= ng the network travel for the persistence. 

 

&= nbsp;

What do you think? Premature opti= mization? Does not make sense? Great idea? Let me know :)

 

 

S

 

=  

 

On Thu, Jan 9, 2014 at 3= :00 PM, Tobias Pazer <t= obiaspazer@gmail.com> wrote:

Hi all,

I have recently sta= rted writing my master thesis with a focus on storm, as we are planning to i= mplement the lambda architecture in our university.

=

As it's s= till not very clear for me where exactly it's worth to dive into, I was hopi= ng one of you might have any suggestions.

I was thinking about a benchmark or something else to syste= matically evaluate and improve the configuration of storm, but I'm not sure = if this is even worth the time.

I think the more experienced = of you definitely have further ideas!

Thanks and regards
Tobias

 

 



--B_3472636057_2920739--