streams-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Danny Sullivan <dsulliv...@hotmail.com>
Subject RE: Using Camel for Routing
Date Wed, 16 Oct 2013 18:12:57 GMT
I think what is happening here is that we have different ideas about the architecture of the
application. 
Here's what I'm thinking:
*All processing of activity would happen BEFORE it's posted to Streams*All activity posted
to Streams is in activitystrea.ms format
The Camel/EIP/OSGi way:
*The user is able to post anything to Streams, doesn't matter what format it's in*The user
can add processing components to Streams that format everything that is input to activitystrea.ms
format, these can be easily plugged in, moved around, or taken out
If we decide to go the second route, I think Camel would be the best way to go. However, I'd
like to make the case for the first approach. The application can be quickly started and tested
by a developer and user with the the first approach. If the user decides they want to experiment
with twitter/facebook data (like Chris suggested) they can write a custom application (which
can be a camel/eip application or even a python, ruby, etc application) that takes in the
twitter/facebook data, formats it to activitystrea.ms format and then POSTs it to Streams.
Also note: the webapp would NOT need to be restarted to add these. I don't think we'd lose
a lot by going this route: in order to add the custom processing components in Camel, a Java
component would still need to be written before it's plugged in to the camel.xml (and I think
you would be tied to Java with this approach). 
Quite simply: We should keep formatting activity separate processing activity
Perhaps I am missing some processing that could happen in between an activity's entrance to
Streams and it's storage? If this is the case, I would make the argument that this processing
should happen on an activities EXIT from Streams (there should be a way for a Subscriber to
specify, in its filters or by some other means, the activity that it wants to see)
-Danny

> Date: Wed, 16 Oct 2013 10:11:23 -0700
> Subject: Re: Using Camel for Routing
> From: chris@cxtsoftware.com
> To: dev@streams.incubator.apache.org
> 
> Jason,
> 
> I agree with you. I guess my vision of streams was a little grander in the
> long run where you could configure the system to accept input from various
> sources (facebook, twitter...) and process them in certain ways. So it
> would have components that would know how to talk with Twitter and convert
> tweets to activities then send it to the right process. So depending on how
> you configure your system, it would create the proper routes from component
> to component to allow processing dynamically which would require a
> framework like Camel (don't think Storm helps us here since it's a compile
> time configuration to my understanding). The nice thing about something
> like camel is it allows you to reconfigure all the routes at runtime. This
> is another reason I liked OSGI support as well as it allows deploying new
> components without restarting the server (webapp). Of course to scale it
> would require being able to deploy the configuration across multiple
> machines which gets fun.
> 
> Maybe my vision is too big for this project but it's where my head has been
> at.
> 
> Chris
> 
> 
> On Wed, Oct 16, 2013 at 7:54 AM, Jason Letourneau
> <jletourneau80@gmail.com>wrote:
> 
> > My only real complaint with this approach is that interjecting new
> > components (processing components) becomes less plug and play without
> > writing Java code in streams - with the Camel routing approach,
> > streams Java is a black box and you just change the camel xml.
> >
> > The primary rationale in using Camel was gaining all of the
> > architectural benefits one gains from  enterprise integration patterns
> > (truth: simplicity is not one and is a tradeoff for flexibility, but
> > generally is considered a universal truth not one relegated to Camel
> > and EIP).
> >
> > Jason
> >
> > On Fri, Oct 11, 2013 at 10:16 AM, Danny Sullivan <dsullivan7@hotmail.com>
> > wrote:
> > > Hey All,
> > > I would like to merge the webservice branch with trunk. I think making
> > streams web service based allows for a better focus on the goal of the
> > application: process activitystrea.ms formatted activity json. While
> > moving to a webservice model would lose us support for many of the use
> > cases Chris mentioned (i.e twitter and facebook), I think the downsides
> > would be mitigated by allowing and encouraging users to write custom
> > appenders that take in twitter/facebook data, format it to
> > activitystrea.ms standards, and POST to the Streams Server. These
> > appenders could even be written in any language as long as the output is
> > activitystrea.ms json. Compared to the trunk branch, I think the web
> > service design employs the "black box" idiom in a much simpler fashion. I
> > would be happy to hear contradicting thoughts and would encourage a
> > discussion on this design switch.
> > > Thanks!
> > > Danny
> > >> From: dsullivan7@hotmail.com
> > >> To: dev@streams.incubator.apache.org
> > >> Subject: RE: Using Camel for Routing
> > >> Date: Fri, 4 Oct 2013 16:07:06 -0400
> > >>
> > >> I wrote up something quickly so I could demo the functionality as a web
> > service as opposed to Camel. The new branch is available here:
> > >> https://svn.apache.org/repos/asf/incubator/streams/branches/webservice/
> > >> note that the new publishing and subscribing urls are:
> > >>
> > http://localhost:8080/streams-web/app/publisherRegisterhttp://localhost:8080/streams-web/app/subscriberRegister
> > >> I used a Spring Web Service only because I was more familiar with it, I
> > would like to research other alternatives (CXF) if it's clear that a web
> > service is a realistic alternative. The main changes to the application are
> > the web controller located here:
> > >>
> > https://svn.apache.org/repos/asf/incubator/streams/branches/webservice/streams-web/src/main/java/org/apache/streams/mvc/controller/StreamsWebController.java
> > >> Which handles the input and output of the application, which were
> > handled by the camel context before. The other main change is the use of
> > the streams-components module located here:
> > >>
> > https://svn.apache.org/repos/asf/incubator/streams/branches/webservice/streams-components/
> > >> This module contains service classes that return the correct output on
> > each request sent to the controller. I reused the Publisher and Subscriber
> > objects as well as the activityConsumerWarehouse and
> > activitySubscriberWarehouse, I think they would be moved to the
> > streams-components package if we want to go this route.
> > >> Overall, I think the flow through the application is much more
> > understandable. Previously, a developer would have to follow an activity
> > through the camelContext.xml as well as through dynamically created routes
> > verses the web service which provides clear entry and exit points to the
> > application. I'd really like to hear feedback on this one, I really don't
> > want to lose any efficiencies that we could be from
> > messaging/osgi/camel/activemq.
> > >> Thanks,
> > >> Danny
> > >>
> > >> > From: dsullivan7@hotmail.com
> > >> > To: dev@streams.incubator.apache.org
> > >> > Subject: RE: Using Camel for Routing
> > >> > Date: Thu, 3 Oct 2013 20:17:13 -0400
> > >> >
> > >> > Thanks for the update Jason. I'll keep the Spring MVC app in a
> > separate branch (hopefully it won't take too long) and then perhaps we can
> > reassess having both component and web container deployment? I think it's a
> > good idea to keep both options in mind especially because this technology
> > is relatively young.
> > >> > Thanks!
> > >> > Danny
> > >> >
> > >> > > Date: Thu, 3 Oct 2013 19:50:59 -0400
> > >> > > Subject: Re: Using Camel for Routing
> > >> > > From: jletourneau80@gmail.com
> > >> > > To: dev@streams.incubator.apache.org
> > >> > >
> > >> > > Because of the osgi container support in addition to deployment
as
> > a web
> > >> > > app - when last we visited this topic - I think the majority
was in
> > favor
> > >> > > of keeping the component style deployment in addition to web
> > container
> > >> > > deployment
> > >> > >
> > >> > > On Thursday, October 3, 2013, Danny Sullivan wrote:
> > >> > >
> > >> > > > Hey all,
> > >> > > > I've been considering taking a stab at moving streams from
it's
> > current
> > >> > > > configuration with Camel to a Spring Web Service. Concerns
over
> > failover
> > >> > > > and load balancing would be mitigated by the Cassandra DB
> > (running on 9160)
> > >> > > > and Storm (which depends on an instance of Zookeeper on
2181).
> > Before I
> > >> > > > start lancing any windmills I thought it'd be nice to get
input
> > on why the
> > >> > > > design choice was made to base the application around Camel?
> > >> > > > I think that tracking the flow of an activity through the
> > application will
> > >> > > > be much easier if we use Spring MVC. I'll set up a new branch
> > using the MVC
> > >> > > > architecture and I'll keep the community updated as I go
along.
> > >> > > > Danny
> > >> >
> > >>
> > >
> >
 		 	   		  
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message