Return-Path: X-Original-To: apmail-streams-dev-archive@minotaur.apache.org Delivered-To: apmail-streams-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id E26E81016F for ; Wed, 23 Oct 2013 15:30:25 +0000 (UTC) Received: (qmail 5941 invoked by uid 500); 23 Oct 2013 15:30:25 -0000 Delivered-To: apmail-streams-dev-archive@streams.apache.org Received: (qmail 5892 invoked by uid 500); 23 Oct 2013 15:30:23 -0000 Mailing-List: contact dev-help@streams.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@streams.incubator.apache.org Delivered-To: mailing list dev@streams.incubator.apache.org Received: (qmail 5860 invoked by uid 99); 23 Oct 2013 15:30:21 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Oct 2013 15:30:21 +0000 X-ASF-Spam-Status: No, hits=3.4 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_REPLY,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of dsullivan7@hotmail.com designates 65.55.90.145 as permitted sender) Received: from [65.55.90.145] (HELO snt0-omc3-s6.snt0.hotmail.com) (65.55.90.145) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Oct 2013 15:30:14 +0000 Received: from SNT149-W12 ([65.55.90.136]) by snt0-omc3-s6.snt0.hotmail.com with Microsoft SMTPSVC(6.0.3790.4675); Wed, 23 Oct 2013 08:29:53 -0700 X-TMN: [QE1mthrAokgicsy5uzdUa0YujVwq47I0] X-Originating-Email: [dsullivan7@hotmail.com] Message-ID: Content-Type: multipart/mixed; boundary="_d544ceaa-1919-431b-b81d-a8aa3263de4c_" From: Danny Sullivan To: "dev@streams.incubator.apache.org" Subject: RE: Using Camel for Routing Date: Wed, 23 Oct 2013 11:29:53 -0400 Importance: Normal In-Reply-To: References: ,,,,,,,,,,,,, MIME-Version: 1.0 X-OriginalArrivalTime: 23 Oct 2013 15:29:53.0362 (UTC) FILETIME=[B877AB20:01CED004] X-Virus-Checked: Checked by ClamAV on apache.org --_d544ceaa-1919-431b-b81d-a8aa3263de4c_ Content-Type: multipart/alternative; boundary="_ac1aa56e-a6e0-4028-8ad1-6b2063b82d1b_" --_ac1aa56e-a6e0-4028-8ad1-6b2063b82d1b_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Hey Guys=2C I'm trying to do a little more research in the efficiency between both the = web-service and the camel implementations of streams. I set up Apache JMete= r (http://jmeter.apache.org/) to do some load testing on both setups. I set= up JMeter to make 1000 POST requests sending valid activitystrea.ms format= ted json to a single publisher url over the course of 5 seconds. I've attac= hed .csv files that give the average response the results=2C but I found th= at the web service was .4 seconds faster on average. My concern is that the= se results may not be consistent for the bigger use cases=2C and Camel may = be faster when deployed to multiple servers. It seems though that for early= use cases=2C the web service would be faster.=20 I do want to make Streams as fast and as scalable as possible and I appreci= ate the patience you guys are giving me through through this big design cha= nge proposal. I would be happy to try a different load test setup and=2C as= always=2C would be happy to hear thoughts on the two implementations. Thanks -Danny > Date: Wed=2C 16 Oct 2013 18:49:02 -0400 > Subject: Re: Using Camel for Routing > From: jletourneau80@gmail.com > To: dev@streams.incubator.apache.org >=20 > I think it depends on how you define 'no problem' - deployment > infrastructure is a big part of that - messaging provides more flexibilit= y > in deployment and capability additions at different layers of the > application vs a single web app and a shotgun approach to evolving it >=20 > I agree it should still be on the table as we define what is most importa= nt > to this community >=20 > On Wednesday=2C October 16=2C 2013=2C Danny Sullivan wrote: >=20 > > I think if we sacrifice scalability in the long run by switching from > > Camel to a web service then we should absolutely continue to use messag= ing > > and Camel. I'm confident=2C however=2C that applications exist that hav= e a web > > service based model that service millions of requests and have no probl= em > > scaling without the help of Camel or any other messaging framework. > > I'm going to keep the web-service branch separate for now=2C but I'd st= ill > > like to keep the option on the table until we explore building scalable > > applications without the use of Camel. Though I can see the benefits of > > using Camel=2C I think the complexity it adds makes it worth taking a l= ook at > > other options. > > -Danny > > > > > Date: Wed=2C 16 Oct 2013 16:26:43 -0400 > > > Subject: Re: Using Camel for Routing > > > From: jletourneau80@gmail.com > > > To: dev@streams.incubator.apache.org > > > > > > Yeah the more I think through this in terms of implications of a sole > > > web app - I am skepticle that it scales to interesting use cases. I > > > am not interested in streams to monitor "a few" facebook friends=2C I= am > > > interested in streams to monitor millions of events and activities > > > generated from web site activity (clicks or tweets whatever) or > > > enterprise assets across the globe. > > > > > > On Wed=2C Oct 16=2C 2013 at 4:15 PM=2C Chris Geer > > wrote: > > > > On Wed=2C Oct 16=2C 2013 at 11:58 AM=2C Danny Sullivan < > > dsullivan7@hotmail.com>wrote: > > > > > > > >> I see the value in Streams as the ability of subscribers to get > > activity > > > >> based on filters. > > > > > > > > > > > > I agree > > > > > > > > > > > >> To be clear=2C I do not want to eliminate processing=2C but do wan= t to > > make > > > >> the focus of processing tailored to the subscribers' filters. My m= ore > > grand > > > >> vision is that only the most relevant activity will be returned to= the > > > >> Subscriber and similar activity will be aggregated into single > > activities > > > >> (this would be a little down the road). This processing would happ= en > > after > > > >> the activity is stored in the DB. > > > >> > > > > > > > > This brings me back to the concern about having it all within one w= eb > > app. > > > > If the webapp accepts inputs=2C aggregates/filters it and manages i= t's > > own > > > > subscribers=2C how you do scale it=2C or provide redundancy? The be= auty of > > > > using a message based system (i.e. Camel...along with Storm possibl= y) > > is > > > > that it allows you to partition the application however you want. S= o > > if you > > > > want to have your ingest web service running on one machine and you= r > > > > subscriber service running somewhere else and a third machine for y= our > > > > really resource intensive aggregation component=2C you can. With th= e web > > app > > > > you could do vertical scaling on one machine=2C or load balancing o= f all > > > > components across multiple machines but it doesn't give you the sav= e > > > > flexibility. > > > > > > > > > > > >> Do you imagine that Streams will come packaged with some processin= g > > > >> components that will format other non-activitystrea.ms formatted > > json? I > > > >> do think it'd be pretty cool if you could fire up Streams=2C input= a > > couple > > > >> Facebook Friends and people you follow on Twitter=2C and have all = of > > that > > > >> activity posted in one place. That might be better suited for a ne= w > > > >> separate module (something like streams-format?) if we decide on t= he > > > >> web-application route. > > > >> > > > > > > > > I think it has to come with something to entice more users. If > > everyone has > > > > to build their own translation to even run "Hello World" it will be= a > > > > larger barrier to entry. > > > > > > > > > > > >> -Danny > > > >> > > > >> > Date: Wed=2C 16 Oct 2013 11:22:51 -0700 > > > >> > Subject: Re: Using Camel for Routing > > > >> > From: chris@cxtsoftware.com > > > >> > To: dev@streams.incubator.apache.org > > > >> > > > > >> > On Wed=2C Oct 16=2C 2013 at 11:12 AM=2C Danny Sullivan < > > dsullivan7@hotmail.com > > > >> >wrote: > > > >> > > > > >> > > I think what is happening here is that we have different ideas > > about > > > >> the > > > >> > > architecture of the application. > > > >> > > Here's what I'm thinking: > > > >> > > *All processing of activity would happen BEFORE it's posted to > > > >> Streams*All > > > >> > > activity posted to Streams is in activitystrea.ms format > > > >> > > The Camel/EIP/OSGi way: > > > >> = --_ac1aa56e-a6e0-4028-8ad1-6b2063b82d1b_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable
Hey Guys=2C

I= 'm trying to do a little more research in the efficiency between both the w= eb-service and the camel implementations of streams. I set up Apache JMeter= (http://jmeter.apache.org/) to do some load= testing on both setups. I set up JMeter to make 1000 POST requests sending= valid activitystrea.ms formatted json to a single publisher url over the c= ourse of 5 seconds. I've attached .csv files that =3Bgive the av= erage response =3Bthe results=2C but I found that the = web service was .4 seconds faster on average. My concern is that these resu= lts may not be consistent for the bigger use cases=2C and Camel may be fast= er when deployed to multiple servers. It seems though that for early use ca= ses=2C the web service would be faster. =3B

I do want to make Streams as = fast and as scalable as possible and I appreciate the patience you guys are= giving me through through this big design change proposal. I would be happ= y to try a different load test setup and=2C as always=2C would be happy to = hear thoughts on the two implementations.

Thanks

-Danny

<= div>>=3B Date: Wed=2C 16 Oct 2013 18:49:02 -0400
>=3B Subject: Re: U= sing Camel for Routing
>=3B From: jletourneau80@gmail.com
>=3B To= : dev@streams.incubator.apache.org
>=3B
>=3B I think it depends = on how you define 'no problem' - deployment
>=3B infrastructure is a b= ig part of that - messaging provides more flexibility
>=3B in deployme= nt and capability additions at different layers of the
>=3B applicatio= n vs a single web app and a shotgun approach to evolving it
>=3B
&= gt=3B I agree it should still be on the table as we define what is most imp= ortant
>=3B to this community
>=3B
>=3B On Wednesday=2C Oct= ober 16=2C 2013=2C Danny Sullivan wrote:
>=3B
>=3B >=3B I thin= k if we sacrifice scalability in the long run by switching from
>=3B &= gt=3B Camel to a web service then we should absolutely continue to use mess= aging
>=3B >=3B and Camel. I'm confident=2C however=2C that applicat= ions exist that have a web
>=3B >=3B service based model that servic= e millions of requests and have no problem
>=3B >=3B scaling without= the help of Camel or any other messaging framework.
>=3B >=3B I'm g= oing to keep the web-service branch separate for now=2C but I'd still
&g= t=3B >=3B like to keep the option on the table until we explore building = scalable
>=3B >=3B applications without the use of Camel. Though I c= an see the benefits of
>=3B >=3B using Camel=2C I think the complexi= ty it adds makes it worth taking a look at
>=3B >=3B other options.<= br>>=3B >=3B -Danny
>=3B >=3B
>=3B >=3B >=3B Date: Wed= =2C 16 Oct 2013 16:26:43 -0400
>=3B >=3B >=3B Subject: Re: Using C= amel for Routing
>=3B >=3B >=3B From: jletourneau80@gmail.com
&= gt=3B >=3B >=3B To: dev@streams.incubator.apache.org
>=3B >=3B &= gt=3B
>=3B >=3B >=3B Yeah the more I think through this in terms o= f implications of a sole
>=3B >=3B >=3B web app - I am skepticle t= hat it scales to interesting use cases. I
>=3B >=3B >=3B am not i= nterested in streams to monitor "a few" facebook friends=2C I am
>=3B = >=3B >=3B interested in streams to monitor millions of events and activ= ities
>=3B >=3B >=3B generated from web site activity (clicks or t= weets whatever) or
>=3B >=3B >=3B enterprise assets across the glo= be.
>=3B >=3B >=3B
>=3B >=3B >=3B On Wed=2C Oct 16=2C 201= 3 at 4:15 PM=2C Chris Geer <=3Bchris@cxtsoftware.com>=3B
>=3B >= =3B wrote:
>=3B >=3B >=3B >=3B On Wed=2C Oct 16=2C 2013 at 11:58= AM=2C Danny Sullivan <=3B
>=3B >=3B dsullivan7@hotmail.com>=3Bw= rote:
>=3B >=3B >=3B >=3B
>=3B >=3B >=3B >=3B>=3B I= see the value in Streams as the ability of subscribers to get
>=3B &g= t=3B activity
>=3B >=3B >=3B >=3B>=3B based on filters.
>= =3B >=3B >=3B >=3B
>=3B >=3B >=3B >=3B
>=3B >=3B &g= t=3B >=3B I agree
>=3B >=3B >=3B >=3B
>=3B >=3B >=3B = >=3B
>=3B >=3B >=3B >=3B>=3B To be clear=2C I do not want to= eliminate processing=2C but do want to
>=3B >=3B make
>=3B >= =3B >=3B >=3B>=3B the focus of processing tailored to the subscribers= ' filters. My more
>=3B >=3B grand
>=3B >=3B >=3B >=3B>= =3B vision is that only the most relevant activity will be returned to the<= br>>=3B >=3B >=3B >=3B>=3B Subscriber and similar activity will b= e aggregated into single
>=3B >=3B activities
>=3B >=3B >= =3B >=3B>=3B (this would be a little down the road). This processing wo= uld happen
>=3B >=3B after
>=3B >=3B >=3B >=3B>=3B the = activity is stored in the DB.
>=3B >=3B >=3B >=3B>=3B
>= =3B >=3B >=3B >=3B
>=3B >=3B >=3B >=3B This brings me back= to the concern about having it all within one web
>=3B >=3B app.>=3B >=3B >=3B >=3B If the webapp accepts inputs=2C aggregates/fil= ters it and manages it's
>=3B >=3B own
>=3B >=3B >=3B >= =3B subscribers=2C how you do scale it=2C or provide redundancy? The beauty= of
>=3B >=3B >=3B >=3B using a message based system (i.e. Camel= ...along with Storm possibly)
>=3B >=3B is
>=3B >=3B >=3B &= gt=3B that it allows you to partition the application however you want. So<= br>>=3B >=3B if you
>=3B >=3B >=3B >=3B want to have your in= gest web service running on one machine and your
>=3B >=3B >=3B &g= t=3B subscriber service running somewhere else and a third machine for your=
>=3B >=3B >=3B >=3B really resource intensive aggregation compo= nent=2C you can. With the web
>=3B >=3B app
>=3B >=3B >=3B = >=3B you could do vertical scaling on one machine=2C or load balancing of= all
>=3B >=3B >=3B >=3B components across multiple machines but= it doesn't give you the save
>=3B >=3B >=3B >=3B flexibility.>=3B >=3B >=3B >=3B
>=3B >=3B >=3B >=3B
>=3B >= =3B >=3B >=3B>=3B Do you imagine that Streams will come packaged with= some processing
>=3B >=3B >=3B >=3B>=3B components that will = format other non-activitystrea.ms formatted
>=3B >=3B json? I
>= =3B >=3B >=3B >=3B>=3B do think it'd be pretty cool if you could fi= re up Streams=2C input a
>=3B >=3B couple
>=3B >=3B >=3B &g= t=3B>=3B Facebook Friends and people you follow on Twitter=2C and have al= l of
>=3B >=3B that
>=3B >=3B >=3B >=3B>=3B activity po= sted in one place. That might be better suited for a new
>=3B >=3B &= gt=3B >=3B>=3B separate module (something like streams-format?) if we d= ecide on the
>=3B >=3B >=3B >=3B>=3B web-application route.>=3B >=3B >=3B >=3B>=3B
>=3B >=3B >=3B >=3B
>=3B= >=3B >=3B >=3B I think it has to come with something to entice more = users. If
>=3B >=3B everyone has
>=3B >=3B >=3B >=3B to b= uild their own translation to even run "Hello World" it will be a
>=3B= >=3B >=3B >=3B larger barrier to entry.
>=3B >=3B >=3B >= =3B
>=3B >=3B >=3B >=3B
>=3B >=3B >=3B >=3B>=3B -Da= nny
>=3B >=3B >=3B >=3B>=3B
>=3B >=3B >=3B >=3B>= =3B >=3B Date: Wed=2C 16 Oct 2013 11:22:51 -0700
>=3B >=3B >=3B = >=3B>=3B >=3B Subject: Re: Using Camel for Routing
>=3B >=3B &= gt=3B >=3B>=3B >=3B From: chris@cxtsoftware.com
>=3B >=3B >= =3B >=3B>=3B >=3B To: dev@streams.incubator.apache.org
>=3B >= =3B >=3B >=3B>=3B >=3B
>=3B >=3B >=3B >=3B>=3B >=3B = On Wed=2C Oct 16=2C 2013 at 11:12 AM=2C Danny Sullivan <=3B
>=3B >= =3B dsullivan7@hotmail.com
>=3B >=3B >=3B >=3B>=3B >=3Bwrote= :
>=3B >=3B >=3B >=3B>=3B >=3B
>=3B >=3B >=3B >= =3B>=3B >=3B >=3B I think what is happening here is that we have diff= erent ideas
>=3B >=3B about
>=3B >=3B >=3B >=3B>=3B the=
>=3B >=3B >=3B >=3B>=3B >=3B >=3B architecture of the app= lication.
>=3B >=3B >=3B >=3B>=3B >=3B >=3B Here's what I'= m thinking:
>=3B >=3B >=3B >=3B>=3B >=3B >=3B *All process= ing of activity would happen BEFORE it's posted to
>=3B >=3B >=3B = >=3B>=3B Streams*All
>=3B >=3B >=3B >=3B>=3B >=3B >=3B= activity posted to Streams is in activitystrea.ms format
>=3B >=3B = >=3B >=3B>=3B >=3B >=3B The Camel/EIP/OSGi way:
>=3B >=3B = >=3B >=3B>=3B
= --_ac1aa56e-a6e0-4028-8ad1-6b2063b82d1b_-- --_d544ceaa-1919-431b-b81d-a8aa3263de4c_--