streams-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Danny Sullivan <dsulliv...@hotmail.com>
Subject RE: Using Camel for Routing
Date Tue, 29 Oct 2013 15:05:55 GMT
That sounds like a good idea Jason, I'll set up the vote!

> Date: Thu, 24 Oct 2013 09:55:10 -0700
> Subject: Re: Using Camel for Routing
> From: chris@cxtsoftware.com
> To: dev@streams.incubator.apache.org
> 
> I agree with Jason personally, flexibility and scalability trumps simple.
> 
> That being said, I honestly don't have enough time right now to spend
> developing on my vision so it's unfair of me to push too hard in that
> direction. If the active developers want to go down the single service path
> maybe that's the approach for now and we can potentially look at
> alternatives down the road.
> 
> Chris
> 
> 
> On Thu, Oct 24, 2013 at 8:08 AM, Jason Letourneau
> <jletourneau80@gmail.com>wrote:
> 
> > I think this comes down to a fundamental question - how flexible vs
> > simple do people want the deployment and configuration of Streams to
> > be?
> >
> > Personally, as I've said, I prefer flexible over simple, but I think
> > the current architecture reaches a happy medium with the WAR
> > deployment option.
> >
> > If the majority wants simple over flexible, then perhaps the web
> > service architecture you've proposed is the way to go.  I think its
> > worth putting to vote.
> >
> > Jason
> >
> > On Wed, Oct 23, 2013 at 11:29 AM, Danny Sullivan <dsullivan7@hotmail.com>
> > wrote:
> > > Hey Guys,
> > >
> > > I'm trying to do a little more research in the efficiency between both
> > the
> > > web-service and the camel implementations of streams. I set up Apache
> > JMeter
> > > (http://jmeter.apache.org/) to do some load testing on both setups. I
> > set up
> > > JMeter to make 1000 POST requests sending valid activitystrea.msformatted
> > > json to a single publisher url over the course of 5 seconds. I've
> > attached
> > > .csv files that give the average response the results, but I found that
> > the
> > > web service was .4 seconds faster on average. My concern is that these
> > > results may not be consistent for the bigger use cases, and Camel may be
> > > faster when deployed to multiple servers. It seems though that for early
> > use
> > > cases, the web service would be faster.
> > >
> > > I do want to make Streams as fast and as scalable as possible and I
> > > appreciate the patience you guys are giving me through through this big
> > > design change proposal. I would be happy to try a different load test
> > setup
> > > and, as always, would be happy to hear thoughts on the two
> > implementations.
> > >
> > > Thanks
> > >
> > > -Danny
> > >
> > >> Date: Wed, 16 Oct 2013 18:49:02 -0400
> > >
> > >> Subject: Re: Using Camel for Routing
> > >> From: jletourneau80@gmail.com
> > >> To: dev@streams.incubator.apache.org
> > >>
> > >> I think it depends on how you define 'no problem' - deployment
> > >> infrastructure is a big part of that - messaging provides more
> > flexibility
> > >> in deployment and capability additions at different layers of the
> > >> application vs a single web app and a shotgun approach to evolving it
> > >>
> > >> I agree it should still be on the table as we define what is most
> > >> important
> > >> to this community
> > >>
> > >> On Wednesday, October 16, 2013, Danny Sullivan wrote:
> > >>
> > >> > I think if we sacrifice scalability in the long run by switching from
> > >> > Camel to a web service then we should absolutely continue to use
> > >> > messaging
> > >> > and Camel. I'm confident, however, that applications exist that have
a
> > >> > web
> > >> > service based model that service millions of requests and have no
> > >> > problem
> > >> > scaling without the help of Camel or any other messaging framework.
> > >> > I'm going to keep the web-service branch separate for now, but I'd
> > still
> > >> > like to keep the option on the table until we explore building
> > scalable
> > >> > applications without the use of Camel. Though I can see the benefits
> > of
> > >> > using Camel, I think the complexity it adds makes it worth taking
a
> > look
> > >> > at
> > >> > other options.
> > >> > -Danny
> > >> >
> > >> > > Date: Wed, 16 Oct 2013 16:26:43 -0400
> > >> > > Subject: Re: Using Camel for Routing
> > >> > > From: jletourneau80@gmail.com
> > >> > > To: dev@streams.incubator.apache.org
> > >> > >
> > >> > > Yeah the more I think through this in terms of implications of
a
> > sole
> > >> > > web app - I am skepticle that it scales to interesting use cases.
I
> > >> > > am not interested in streams to monitor "a few" facebook friends,
I
> > am
> > >> > > interested in streams to monitor millions of events and activities
> > >> > > generated from web site activity (clicks or tweets whatever)
or
> > >> > > enterprise assets across the globe.
> > >> > >
> > >> > > On Wed, Oct 16, 2013 at 4:15 PM, Chris Geer <chris@cxtsoftware.com>
> > >> > wrote:
> > >> > > > On Wed, Oct 16, 2013 at 11:58 AM, Danny Sullivan <
> > >> > dsullivan7@hotmail.com>wrote:
> > >> > > >
> > >> > > >> I see the value in Streams as the ability of subscribers
to get
> > >> > activity
> > >> > > >> based on filters.
> > >> > > >
> > >> > > >
> > >> > > > I agree
> > >> > > >
> > >> > > >
> > >> > > >> To be clear, I do not want to eliminate processing,
but do want
> > to
> > >> > make
> > >> > > >> the focus of processing tailored to the subscribers'
filters. My
> > >> > > >> more
> > >> > grand
> > >> > > >> vision is that only the most relevant activity will
be returned
> > to
> > >> > > >> the
> > >> > > >> Subscriber and similar activity will be aggregated into
single
> > >> > activities
> > >> > > >> (this would be a little down the road). This processing
would
> > >> > > >> happen
> > >> > after
> > >> > > >> the activity is stored in the DB.
> > >> > > >>
> > >> > > >
> > >> > > > This brings me back to the concern about having it all within
one
> > >> > > > web
> > >> > app.
> > >> > > > If the webapp accepts inputs, aggregates/filters it and
manages
> > it's
> > >> > own
> > >> > > > subscribers, how you do scale it, or provide redundancy?
The
> > beauty
> > >> > > > of
> > >> > > > using a message based system (i.e. Camel...along with Storm
> > >> > > > possibly)
> > >> > is
> > >> > > > that it allows you to partition the application however
you want.
> > So
> > >> > if you
> > >> > > > want to have your ingest web service running on one machine
and
> > your
> > >> > > > subscriber service running somewhere else and a third machine
for
> > >> > > > your
> > >> > > > really resource intensive aggregation component, you can.
With the
> > >> > > > web
> > >> > app
> > >> > > > you could do vertical scaling on one machine, or load balancing
of
> > >> > > > all
> > >> > > > components across multiple machines but it doesn't give
you the
> > save
> > >> > > > flexibility.
> > >> > > >
> > >> > > >
> > >> > > >> Do you imagine that Streams will come packaged with
some
> > processing
> > >> > > >> components that will format other non-activitystrea.ms
formatted
> > >> > json? I
> > >> > > >> do think it'd be pretty cool if you could fire up Streams,
input
> > a
> > >> > couple
> > >> > > >> Facebook Friends and people you follow on Twitter, and
have all
> > of
> > >> > that
> > >> > > >> activity posted in one place. That might be better suited
for a
> > new
> > >> > > >> separate module (something like streams-format?) if
we decide on
> > >> > > >> the
> > >> > > >> web-application route.
> > >> > > >>
> > >> > > >
> > >> > > > I think it has to come with something to entice more users.
If
> > >> > everyone has
> > >> > > > to build their own translation to even run "Hello World"
it will
> > be
> > >> > > > a
> > >> > > > larger barrier to entry.
> > >> > > >
> > >> > > >
> > >> > > >> -Danny
> > >> > > >>
> > >> > > >> > Date: Wed, 16 Oct 2013 11:22:51 -0700
> > >> > > >> > Subject: Re: Using Camel for Routing
> > >> > > >> > From: chris@cxtsoftware.com
> > >> > > >> > To: dev@streams.incubator.apache.org
> > >> > > >> >
> > >> > > >> > On Wed, Oct 16, 2013 at 11:12 AM, Danny Sullivan
<
> > >> > dsullivan7@hotmail.com
> > >> > > >> >wrote:
> > >> > > >> >
> > >> > > >> > > I think what is happening here is that we
have different
> > ideas
> > >> > about
> > >> > > >> the
> > >> > > >> > > architecture of the application.
> > >> > > >> > > Here's what I'm thinking:
> > >> > > >> > > *All processing of activity would happen BEFORE
it's posted
> > to
> > >> > > >> Streams*All
> > >> > > >> > > activity posted to Streams is in activitystrea.ms
format
> > >> > > >> > > The Camel/EIP/OSGi way:
> > >> > > >>
> >
 		 	   		  
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message