streams-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Danny Sullivan <dsulliv...@hotmail.com>
Subject RE: Using Camel for Routing
Date Wed, 23 Oct 2013 15:29:53 GMT
Hey Guys,
I'm trying to do a little more research in the efficiency between both the web-service and
the camel implementations of streams. I set up Apache JMeter (http://jmeter.apache.org/) to
do some load testing on both setups. I set up JMeter to make 1000 POST requests sending valid
activitystrea.ms formatted json to a single publisher url over the course of 5 seconds. I've
attached .csv files that give the average response the results, but I found that the web service
was .4 seconds faster on average. My concern is that these results may not be consistent for
the bigger use cases, and Camel may be faster when deployed to multiple servers. It seems
though that for early use cases, the web service would be faster. 
I do want to make Streams as fast and as scalable as possible and I appreciate the patience
you guys are giving me through through this big design change proposal. I would be happy to
try a different load test setup and, as always, would be happy to hear thoughts on the two
implementations.
Thanks
-Danny
> Date: Wed, 16 Oct 2013 18:49:02 -0400
> Subject: Re: Using Camel for Routing
> From: jletourneau80@gmail.com
> To: dev@streams.incubator.apache.org
> 
> I think it depends on how you define 'no problem' - deployment
> infrastructure is a big part of that - messaging provides more flexibility
> in deployment and capability additions at different layers of the
> application vs a single web app and a shotgun approach to evolving it
> 
> I agree it should still be on the table as we define what is most important
> to this community
> 
> On Wednesday, October 16, 2013, Danny Sullivan wrote:
> 
> > I think if we sacrifice scalability in the long run by switching from
> > Camel to a web service then we should absolutely continue to use messaging
> > and Camel. I'm confident, however, that applications exist that have a web
> > service based model that service millions of requests and have no problem
> > scaling without the help of Camel or any other messaging framework.
> > I'm going to keep the web-service branch separate for now, but I'd still
> > like to keep the option on the table until we explore building scalable
> > applications without the use of Camel. Though I can see the benefits of
> > using Camel, I think the complexity it adds makes it worth taking a look at
> > other options.
> > -Danny
> >
> > > Date: Wed, 16 Oct 2013 16:26:43 -0400
> > > Subject: Re: Using Camel for Routing
> > > From: jletourneau80@gmail.com
> > > To: dev@streams.incubator.apache.org
> > >
> > > Yeah the more I think through this in terms of implications of a sole
> > > web app - I am skepticle that it scales to interesting use cases.  I
> > > am not interested in streams to monitor "a few" facebook friends, I am
> > > interested in streams to monitor millions of events and activities
> > > generated from web site activity (clicks or tweets whatever) or
> > > enterprise assets across the globe.
> > >
> > > On Wed, Oct 16, 2013 at 4:15 PM, Chris Geer <chris@cxtsoftware.com>
> > wrote:
> > > > On Wed, Oct 16, 2013 at 11:58 AM, Danny Sullivan <
> > dsullivan7@hotmail.com>wrote:
> > > >
> > > >> I see the value in Streams as the ability of subscribers to get
> > activity
> > > >> based on filters.
> > > >
> > > >
> > > > I agree
> > > >
> > > >
> > > >> To be clear, I do not want to eliminate processing, but do want to
> > make
> > > >> the focus of processing tailored to the subscribers' filters. My more
> > grand
> > > >> vision is that only the most relevant activity will be returned to
the
> > > >> Subscriber and similar activity will be aggregated into single
> > activities
> > > >> (this would be a little down the road). This processing would happen
> > after
> > > >> the activity is stored in the DB.
> > > >>
> > > >
> > > > This brings me back to the concern about having it all within one web
> > app.
> > > > If the webapp accepts inputs, aggregates/filters it and manages it's
> > own
> > > > subscribers, how you do scale it, or provide redundancy? The beauty of
> > > > using a message based system (i.e. Camel...along with Storm possibly)
> > is
> > > > that it allows you to partition the application however you want. So
> > if you
> > > > want to have your ingest web service running on one machine and your
> > > > subscriber service running somewhere else and a third machine for your
> > > > really resource intensive aggregation component, you can. With the web
> > app
> > > > you could do vertical scaling on one machine, or load balancing of all
> > > > components across multiple machines but it doesn't give you the save
> > > > flexibility.
> > > >
> > > >
> > > >> Do you imagine that Streams will come packaged with some processing
> > > >> components that will format other non-activitystrea.ms formatted
> > json? I
> > > >> do think it'd be pretty cool if you could fire up Streams, input a
> > couple
> > > >> Facebook Friends and people you follow on Twitter, and have all of
> > that
> > > >> activity posted in one place. That might be better suited for a new
> > > >> separate module (something like streams-format?) if we decide on the
> > > >> web-application route.
> > > >>
> > > >
> > > > I think it has to come with something to entice more users. If
> > everyone has
> > > > to build their own translation to even run "Hello World" it will be a
> > > > larger barrier to entry.
> > > >
> > > >
> > > >> -Danny
> > > >>
> > > >> > Date: Wed, 16 Oct 2013 11:22:51 -0700
> > > >> > Subject: Re: Using Camel for Routing
> > > >> > From: chris@cxtsoftware.com
> > > >> > To: dev@streams.incubator.apache.org
> > > >> >
> > > >> > On Wed, Oct 16, 2013 at 11:12 AM, Danny Sullivan <
> > dsullivan7@hotmail.com
> > > >> >wrote:
> > > >> >
> > > >> > > I think what is happening here is that we have different
ideas
> > about
> > > >> the
> > > >> > > architecture of the application.
> > > >> > > Here's what I'm thinking:
> > > >> > > *All processing of activity would happen BEFORE it's posted
to
> > > >> Streams*All
> > > >> > > activity posted to Streams is in activitystrea.ms format
> > > >> > > The Camel/EIP/OSGi way:
> > > >>
 		 	   		  
Mime
  • Unnamed multipart/mixed (inline, None, 0 bytes)
View raw message