streams-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Danny Sullivan <dsulliv...@hotmail.com>
Subject RE: Using Camel for Routing
Date Wed, 16 Oct 2013 22:13:52 GMT
I think if we sacrifice scalability in the long run by switching from Camel to a web service
then we should absolutely continue to use messaging and Camel. I'm confident, however, that
applications exist that have a web service based model that service millions of requests and
have no problem scaling without the help of Camel or any other messaging framework. 
I'm going to keep the web-service branch separate for now, but I'd still like to keep the
option on the table until we explore building scalable applications without the use of Camel.
Though I can see the benefits of using Camel, I think the complexity it adds makes it worth
taking a look at other options.
-Danny 

> Date: Wed, 16 Oct 2013 16:26:43 -0400
> Subject: Re: Using Camel for Routing
> From: jletourneau80@gmail.com
> To: dev@streams.incubator.apache.org
> 
> Yeah the more I think through this in terms of implications of a sole
> web app - I am skepticle that it scales to interesting use cases.  I
> am not interested in streams to monitor "a few" facebook friends, I am
> interested in streams to monitor millions of events and activities
> generated from web site activity (clicks or tweets whatever) or
> enterprise assets across the globe.
> 
> On Wed, Oct 16, 2013 at 4:15 PM, Chris Geer <chris@cxtsoftware.com> wrote:
> > On Wed, Oct 16, 2013 at 11:58 AM, Danny Sullivan <dsullivan7@hotmail.com>wrote:
> >
> >> I see the value in Streams as the ability of subscribers to get activity
> >> based on filters.
> >
> >
> > I agree
> >
> >
> >> To be clear, I do not want to eliminate processing, but do want to make
> >> the focus of processing tailored to the subscribers' filters. My more grand
> >> vision is that only the most relevant activity will be returned to the
> >> Subscriber and similar activity will be aggregated into single activities
> >> (this would be a little down the road). This processing would happen after
> >> the activity is stored in the DB.
> >>
> >
> > This brings me back to the concern about having it all within one web app.
> > If the webapp accepts inputs, aggregates/filters it and manages it's own
> > subscribers, how you do scale it, or provide redundancy? The beauty of
> > using a message based system (i.e. Camel...along with Storm possibly) is
> > that it allows you to partition the application however you want. So if you
> > want to have your ingest web service running on one machine and your
> > subscriber service running somewhere else and a third machine for your
> > really resource intensive aggregation component, you can. With the web app
> > you could do vertical scaling on one machine, or load balancing of all
> > components across multiple machines but it doesn't give you the save
> > flexibility.
> >
> >
> >> Do you imagine that Streams will come packaged with some processing
> >> components that will format other non-activitystrea.ms formatted json? I
> >> do think it'd be pretty cool if you could fire up Streams, input a couple
> >> Facebook Friends and people you follow on Twitter, and have all of that
> >> activity posted in one place. That might be better suited for a new
> >> separate module (something like streams-format?) if we decide on the
> >> web-application route.
> >>
> >
> > I think it has to come with something to entice more users. If everyone has
> > to build their own translation to even run "Hello World" it will be a
> > larger barrier to entry.
> >
> >
> >> -Danny
> >>
> >> > Date: Wed, 16 Oct 2013 11:22:51 -0700
> >> > Subject: Re: Using Camel for Routing
> >> > From: chris@cxtsoftware.com
> >> > To: dev@streams.incubator.apache.org
> >> >
> >> > On Wed, Oct 16, 2013 at 11:12 AM, Danny Sullivan <dsullivan7@hotmail.com
> >> >wrote:
> >> >
> >> > > I think what is happening here is that we have different ideas about
> >> the
> >> > > architecture of the application.
> >> > > Here's what I'm thinking:
> >> > > *All processing of activity would happen BEFORE it's posted to
> >> Streams*All
> >> > > activity posted to Streams is in activitystrea.ms format
> >> > > The Camel/EIP/OSGi way:
> >> > > *The user is able to post anything to Streams, doesn't matter what
> >> format
> >> > > it's in*The user can add processing components to Streams that format
> >> > > everything that is input to activitystrea.ms format, these can be
> >> easily
> >> > > plugged in, moved around, or taken out
> >> > > If we decide to go the second route, I think Camel would be the best
> >> way
> >> > > to go. However, I'd like to make the case for the first approach.
The
> >> > > application can be quickly started and tested by a developer and user
> >> with
> >> > > the the first approach. If the user decides they want to experiment
> >> with
> >> > > twitter/facebook data (like Chris suggested) they can write a custom
> >> > > application (which can be a camel/eip application or even a python,
> >> ruby,
> >> > > etc application) that takes in the twitter/facebook data, formats
it to
> >> > > activitystrea.ms format and then POSTs it to Streams. Also note: the
> >> > > webapp would NOT need to be restarted to add these. I don't think
we'd
> >> lose
> >> > > a lot by going this route: in order to add the custom processing
> >> components
> >> > > in Camel, a Java component would still need to be written before it's
> >> > > plugged in to the camel.xml (and I think you would be tied to Java
with
> >> > > this approach).
> >> > > Quite simply: We should keep formatting activity separate processing
> >> > > activity
> >> > > Perhaps I am missing some processing that could happen in between
an
> >> > > activity's entrance to Streams and it's storage? If this is the case,
I
> >> > > would make the argument that this processing should happen on an
> >> activities
> >> > > EXIT from Streams (there should be a way for a Subscriber to specify,
> >> in
> >> > > its filters or by some other means, the activity that it wants to
see)
> >> > >
> >> >
> >> > Danny, what do you see as the scope for Streams? If you eliminate data
> >> > input in non-activitystrea.ms format and eliminate processing data what
> >> > does it do, just store each message (unmodified) and output it to
> >> > subscribers?
> >> >
> >> >
> >> > > -Danny
> >> > >
> >> > > > Date: Wed, 16 Oct 2013 10:11:23 -0700
> >> > > > Subject: Re: Using Camel for Routing
> >> > > > From: chris@cxtsoftware.com
> >> > > > To: dev@streams.incubator.apache.org
> >> > > >
> >> > > > Jason,
> >> > > >
> >> > > > I agree with you. I guess my vision of streams was a little grander
> >> in
> >> > > the
> >> > > > long run where you could configure the system to accept input
from
> >> > > various
> >> > > > sources (facebook, twitter...) and process them in certain ways.
So
> >> it
> >> > > > would have components that would know how to talk with Twitter
and
> >> > > convert
> >> > > > tweets to activities then send it to the right process. So depending
> >> on
> >> > > how
> >> > > > you configure your system, it would create the proper routes
from
> >> > > component
> >> > > > to component to allow processing dynamically which would require
a
> >> > > > framework like Camel (don't think Storm helps us here since it's
a
> >> > > compile
> >> > > > time configuration to my understanding). The nice thing about
> >> something
> >> > > > like camel is it allows you to reconfigure all the routes at
runtime.
> >> > > This
> >> > > > is another reason I liked OSGI support as well as it allows
> >> deploying new
> >> > > > components without restarting the server (webapp). Of course
to
> >> scale it
> >> > > > would require being able to deploy the configuration across multiple
> >> > > > machines which gets fun.
> >> > > >
> >> > > > Maybe my vision is too big for this project but it's where my
head
> >> has
> >> > > been
> >> > > > at.
> >> > > >
> >> > > > Chris
> >> > > >
> >> > > >
> >> > > > On Wed, Oct 16, 2013 at 7:54 AM, Jason Letourneau
> >> > > > <jletourneau80@gmail.com>wrote:
> >> > > >
> >> > > > > My only real complaint with this approach is that interjecting
new
> >> > > > > components (processing components) becomes less plug and
play
> >> without
> >> > > > > writing Java code in streams - with the Camel routing approach,
> >> > > > > streams Java is a black box and you just change the camel
xml.
> >> > > > >
> >> > > > > The primary rationale in using Camel was gaining all of
the
> >> > > > > architectural benefits one gains from  enterprise integration
> >> patterns
> >> > > > > (truth: simplicity is not one and is a tradeoff for flexibility,
> >> but
> >> > > > > generally is considered a universal truth not one relegated
to
> >> Camel
> >> > > > > and EIP).
> >> > > > >
> >> > > > > Jason
> >> > > > >
> >> > > > > On Fri, Oct 11, 2013 at 10:16 AM, Danny Sullivan <
> >> > > dsullivan7@hotmail.com>
> >> > > > > wrote:
> >> > > > > > Hey All,
> >> > > > > > I would like to merge the webservice branch with trunk.
I think
> >> > > making
> >> > > > > streams web service based allows for a better focus on the
goal of
> >> the
> >> > > > > application: process activitystrea.ms formatted activity
json.
> >> While
> >> > > > > moving to a webservice model would lose us support for many
of the
> >> use
> >> > > > > cases Chris mentioned (i.e twitter and facebook), I think
the
> >> downsides
> >> > > > > would be mitigated by allowing and encouraging users to
write
> >> custom
> >> > > > > appenders that take in twitter/facebook data, format it
to
> >> > > > > activitystrea.ms standards, and POST to the Streams Server.
These
> >> > > > > appenders could even be written in any language as long
as the
> >> output
> >> > > is
> >> > > > > activitystrea.ms json. Compared to the trunk branch, I think
the
> >> web
> >> > > > > service design employs the "black box" idiom in a much simpler
> >> > > fashion. I
> >> > > > > would be happy to hear contradicting thoughts and would
encourage a
> >> > > > > discussion on this design switch.
> >> > > > > > Thanks!
> >> > > > > > Danny
> >> > > > > >> From: dsullivan7@hotmail.com
> >> > > > > >> To: dev@streams.incubator.apache.org
> >> > > > > >> Subject: RE: Using Camel for Routing
> >> > > > > >> Date: Fri, 4 Oct 2013 16:07:06 -0400
> >> > > > > >>
> >> > > > > >> I wrote up something quickly so I could demo the
functionality
> >> as a
> >> > > web
> >> > > > > service as opposed to Camel. The new branch is available
here:
> >> > > > > >>
> >> > >
> >> https://svn.apache.org/repos/asf/incubator/streams/branches/webservice/
> >> > > > > >> note that the new publishing and subscribing urls
are:
> >> > > > > >>
> >> > > > >
> >> > >
> >> http://localhost:8080/streams-web/app/publisherRegisterhttp://localhost:8080/streams-web/app/subscriberRegister
> >> > > > > >> I used a Spring Web Service only because I was
more familiar
> >> with
> >> > > it, I
> >> > > > > would like to research other alternatives (CXF) if it's
clear that
> >> a
> >> > > web
> >> > > > > service is a realistic alternative. The main changes to
the
> >> > > application are
> >> > > > > the web controller located here:
> >> > > > > >>
> >> > > > >
> >> > >
> >> https://svn.apache.org/repos/asf/incubator/streams/branches/webservice/streams-web/src/main/java/org/apache/streams/mvc/controller/StreamsWebController.java
> >> > > > > >> Which handles the input and output of the application,
which
> >> were
> >> > > > > handled by the camel context before. The other main change
is the
> >> use
> >> > > of
> >> > > > > the streams-components module located here:
> >> > > > > >>
> >> > > > >
> >> > >
> >> https://svn.apache.org/repos/asf/incubator/streams/branches/webservice/streams-components/
> >> > > > > >> This module contains service classes that return
the correct
> >> output
> >> > > on
> >> > > > > each request sent to the controller. I reused the Publisher
and
> >> > > Subscriber
> >> > > > > objects as well as the activityConsumerWarehouse and
> >> > > > > activitySubscriberWarehouse, I think they would be moved
to the
> >> > > > > streams-components package if we want to go this route.
> >> > > > > >> Overall, I think the flow through the application
is much more
> >> > > > > understandable. Previously, a developer would have to follow
an
> >> > > activity
> >> > > > > through the camelContext.xml as well as through dynamically
created
> >> > > routes
> >> > > > > verses the web service which provides clear entry and exit
points
> >> to
> >> > > the
> >> > > > > application. I'd really like to hear feedback on this one,
I really
> >> > > don't
> >> > > > > want to lose any efficiencies that we could be from
> >> > > > > messaging/osgi/camel/activemq.
> >> > > > > >> Thanks,
> >> > > > > >> Danny
> >> > > > > >>
> >> > > > > >> > From: dsullivan7@hotmail.com
> >> > > > > >> > To: dev@streams.incubator.apache.org
> >> > > > > >> > Subject: RE: Using Camel for Routing
> >> > > > > >> > Date: Thu, 3 Oct 2013 20:17:13 -0400
> >> > > > > >> >
> >> > > > > >> > Thanks for the update Jason. I'll keep the
Spring MVC app in a
> >> > > > > separate branch (hopefully it won't take too long) and then
> >> perhaps we
> >> > > can
> >> > > > > reassess having both component and web container deployment?
I
> >> think
> >> > > it's a
> >> > > > > good idea to keep both options in mind especially because
this
> >> > > technology
> >> > > > > is relatively young.
> >> > > > > >> > Thanks!
> >> > > > > >> > Danny
> >> > > > > >> >
> >> > > > > >> > > Date: Thu, 3 Oct 2013 19:50:59 -0400
> >> > > > > >> > > Subject: Re: Using Camel for Routing
> >> > > > > >> > > From: jletourneau80@gmail.com
> >> > > > > >> > > To: dev@streams.incubator.apache.org
> >> > > > > >> > >
> >> > > > > >> > > Because of the osgi container support
in addition to
> >> deployment
> >> > > as
> >> > > > > a web
> >> > > > > >> > > app - when last we visited this topic
- I think the majority
> >> > > was in
> >> > > > > favor
> >> > > > > >> > > of keeping the component style deployment
in addition to web
> >> > > > > container
> >> > > > > >> > > deployment
> >> > > > > >> > >
> >> > > > > >> > > On Thursday, October 3, 2013, Danny Sullivan
wrote:
> >> > > > > >> > >
> >> > > > > >> > > > Hey all,
> >> > > > > >> > > > I've been considering taking a stab
at moving streams from
> >> > > it's
> >> > > > > current
> >> > > > > >> > > > configuration with Camel to a Spring
Web Service. Concerns
> >> > > over
> >> > > > > failover
> >> > > > > >> > > > and load balancing would be mitigated
by the Cassandra DB
> >> > > > > (running on 9160)
> >> > > > > >> > > > and Storm (which depends on an instance
of Zookeeper on
> >> 2181).
> >> > > > > Before I
> >> > > > > >> > > > start lancing any windmills I thought
it'd be nice to get
> >> > > input
> >> > > > > on why the
> >> > > > > >> > > > design choice was made to base the
application around
> >> Camel?
> >> > > > > >> > > > I think that tracking the flow of
an activity through the
> >> > > > > application will
> >> > > > > >> > > > be much easier if we use Spring
MVC. I'll set up a new
> >> branch
> >> > > > > using the MVC
> >> > > > > >> > > > architecture and I'll keep the community
updated as I go
> >> > > along.
> >> > > > > >> > > > Danny
> >> > > > > >> >
> >> > > > > >>
> >> > > > > >
> >> > > > >
> >> > >
> >> > >
> >>
> >>
 		 	   		  
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message