streams-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason Letourneau <jletournea...@gmail.com>
Subject Re: Using Camel for Routing
Date Thu, 24 Oct 2013 15:08:55 GMT
I think this comes down to a fundamental question - how flexible vs
simple do people want the deployment and configuration of Streams to
be?

Personally, as I've said, I prefer flexible over simple, but I think
the current architecture reaches a happy medium with the WAR
deployment option.

If the majority wants simple over flexible, then perhaps the web
service architecture you've proposed is the way to go.  I think its
worth putting to vote.

Jason

On Wed, Oct 23, 2013 at 11:29 AM, Danny Sullivan <dsullivan7@hotmail.com> wrote:
> Hey Guys,
>
> I'm trying to do a little more research in the efficiency between both the
> web-service and the camel implementations of streams. I set up Apache JMeter
> (http://jmeter.apache.org/) to do some load testing on both setups. I set up
> JMeter to make 1000 POST requests sending valid activitystrea.ms formatted
> json to a single publisher url over the course of 5 seconds. I've attached
> .csv files that give the average response the results, but I found that the
> web service was .4 seconds faster on average. My concern is that these
> results may not be consistent for the bigger use cases, and Camel may be
> faster when deployed to multiple servers. It seems though that for early use
> cases, the web service would be faster.
>
> I do want to make Streams as fast and as scalable as possible and I
> appreciate the patience you guys are giving me through through this big
> design change proposal. I would be happy to try a different load test setup
> and, as always, would be happy to hear thoughts on the two implementations.
>
> Thanks
>
> -Danny
>
>> Date: Wed, 16 Oct 2013 18:49:02 -0400
>
>> Subject: Re: Using Camel for Routing
>> From: jletourneau80@gmail.com
>> To: dev@streams.incubator.apache.org
>>
>> I think it depends on how you define 'no problem' - deployment
>> infrastructure is a big part of that - messaging provides more flexibility
>> in deployment and capability additions at different layers of the
>> application vs a single web app and a shotgun approach to evolving it
>>
>> I agree it should still be on the table as we define what is most
>> important
>> to this community
>>
>> On Wednesday, October 16, 2013, Danny Sullivan wrote:
>>
>> > I think if we sacrifice scalability in the long run by switching from
>> > Camel to a web service then we should absolutely continue to use
>> > messaging
>> > and Camel. I'm confident, however, that applications exist that have a
>> > web
>> > service based model that service millions of requests and have no
>> > problem
>> > scaling without the help of Camel or any other messaging framework.
>> > I'm going to keep the web-service branch separate for now, but I'd still
>> > like to keep the option on the table until we explore building scalable
>> > applications without the use of Camel. Though I can see the benefits of
>> > using Camel, I think the complexity it adds makes it worth taking a look
>> > at
>> > other options.
>> > -Danny
>> >
>> > > Date: Wed, 16 Oct 2013 16:26:43 -0400
>> > > Subject: Re: Using Camel for Routing
>> > > From: jletourneau80@gmail.com
>> > > To: dev@streams.incubator.apache.org
>> > >
>> > > Yeah the more I think through this in terms of implications of a sole
>> > > web app - I am skepticle that it scales to interesting use cases. I
>> > > am not interested in streams to monitor "a few" facebook friends, I am
>> > > interested in streams to monitor millions of events and activities
>> > > generated from web site activity (clicks or tweets whatever) or
>> > > enterprise assets across the globe.
>> > >
>> > > On Wed, Oct 16, 2013 at 4:15 PM, Chris Geer <chris@cxtsoftware.com>
>> > wrote:
>> > > > On Wed, Oct 16, 2013 at 11:58 AM, Danny Sullivan <
>> > dsullivan7@hotmail.com>wrote:
>> > > >
>> > > >> I see the value in Streams as the ability of subscribers to get
>> > activity
>> > > >> based on filters.
>> > > >
>> > > >
>> > > > I agree
>> > > >
>> > > >
>> > > >> To be clear, I do not want to eliminate processing, but do want
to
>> > make
>> > > >> the focus of processing tailored to the subscribers' filters.
My
>> > > >> more
>> > grand
>> > > >> vision is that only the most relevant activity will be returned
to
>> > > >> the
>> > > >> Subscriber and similar activity will be aggregated into single
>> > activities
>> > > >> (this would be a little down the road). This processing would
>> > > >> happen
>> > after
>> > > >> the activity is stored in the DB.
>> > > >>
>> > > >
>> > > > This brings me back to the concern about having it all within one
>> > > > web
>> > app.
>> > > > If the webapp accepts inputs, aggregates/filters it and manages it's
>> > own
>> > > > subscribers, how you do scale it, or provide redundancy? The beauty
>> > > > of
>> > > > using a message based system (i.e. Camel...along with Storm
>> > > > possibly)
>> > is
>> > > > that it allows you to partition the application however you want.
So
>> > if you
>> > > > want to have your ingest web service running on one machine and your
>> > > > subscriber service running somewhere else and a third machine for
>> > > > your
>> > > > really resource intensive aggregation component, you can. With the
>> > > > web
>> > app
>> > > > you could do vertical scaling on one machine, or load balancing of
>> > > > all
>> > > > components across multiple machines but it doesn't give you the save
>> > > > flexibility.
>> > > >
>> > > >
>> > > >> Do you imagine that Streams will come packaged with some processing
>> > > >> components that will format other non-activitystrea.ms formatted
>> > json? I
>> > > >> do think it'd be pretty cool if you could fire up Streams, input
a
>> > couple
>> > > >> Facebook Friends and people you follow on Twitter, and have all
of
>> > that
>> > > >> activity posted in one place. That might be better suited for
a new
>> > > >> separate module (something like streams-format?) if we decide
on
>> > > >> the
>> > > >> web-application route.
>> > > >>
>> > > >
>> > > > I think it has to come with something to entice more users. If
>> > everyone has
>> > > > to build their own translation to even run "Hello World" it will be
>> > > > a
>> > > > larger barrier to entry.
>> > > >
>> > > >
>> > > >> -Danny
>> > > >>
>> > > >> > Date: Wed, 16 Oct 2013 11:22:51 -0700
>> > > >> > Subject: Re: Using Camel for Routing
>> > > >> > From: chris@cxtsoftware.com
>> > > >> > To: dev@streams.incubator.apache.org
>> > > >> >
>> > > >> > On Wed, Oct 16, 2013 at 11:12 AM, Danny Sullivan <
>> > dsullivan7@hotmail.com
>> > > >> >wrote:
>> > > >> >
>> > > >> > > I think what is happening here is that we have different
ideas
>> > about
>> > > >> the
>> > > >> > > architecture of the application.
>> > > >> > > Here's what I'm thinking:
>> > > >> > > *All processing of activity would happen BEFORE it's
posted to
>> > > >> Streams*All
>> > > >> > > activity posted to Streams is in activitystrea.ms format
>> > > >> > > The Camel/EIP/OSGi way:
>> > > >>

Mime
View raw message