streams-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Geer <ch...@cxtsoftware.com>
Subject Re: Using Camel for Routing
Date Wed, 16 Oct 2013 18:22:51 GMT
On Wed, Oct 16, 2013 at 11:12 AM, Danny Sullivan <dsullivan7@hotmail.com>wrote:

> I think what is happening here is that we have different ideas about the
> architecture of the application.
> Here's what I'm thinking:
> *All processing of activity would happen BEFORE it's posted to Streams*All
> activity posted to Streams is in activitystrea.ms format
> The Camel/EIP/OSGi way:
> *The user is able to post anything to Streams, doesn't matter what format
> it's in*The user can add processing components to Streams that format
> everything that is input to activitystrea.ms format, these can be easily
> plugged in, moved around, or taken out
> If we decide to go the second route, I think Camel would be the best way
> to go. However, I'd like to make the case for the first approach. The
> application can be quickly started and tested by a developer and user with
> the the first approach. If the user decides they want to experiment with
> twitter/facebook data (like Chris suggested) they can write a custom
> application (which can be a camel/eip application or even a python, ruby,
> etc application) that takes in the twitter/facebook data, formats it to
> activitystrea.ms format and then POSTs it to Streams. Also note: the
> webapp would NOT need to be restarted to add these. I don't think we'd lose
> a lot by going this route: in order to add the custom processing components
> in Camel, a Java component would still need to be written before it's
> plugged in to the camel.xml (and I think you would be tied to Java with
> this approach).
> Quite simply: We should keep formatting activity separate processing
> activity
> Perhaps I am missing some processing that could happen in between an
> activity's entrance to Streams and it's storage? If this is the case, I
> would make the argument that this processing should happen on an activities
> EXIT from Streams (there should be a way for a Subscriber to specify, in
> its filters or by some other means, the activity that it wants to see)
>

Danny, what do you see as the scope for Streams? If you eliminate data
input in non-activitystrea.ms format and eliminate processing data what
does it do, just store each message (unmodified) and output it to
subscribers?


> -Danny
>
> > Date: Wed, 16 Oct 2013 10:11:23 -0700
> > Subject: Re: Using Camel for Routing
> > From: chris@cxtsoftware.com
> > To: dev@streams.incubator.apache.org
> >
> > Jason,
> >
> > I agree with you. I guess my vision of streams was a little grander in
> the
> > long run where you could configure the system to accept input from
> various
> > sources (facebook, twitter...) and process them in certain ways. So it
> > would have components that would know how to talk with Twitter and
> convert
> > tweets to activities then send it to the right process. So depending on
> how
> > you configure your system, it would create the proper routes from
> component
> > to component to allow processing dynamically which would require a
> > framework like Camel (don't think Storm helps us here since it's a
> compile
> > time configuration to my understanding). The nice thing about something
> > like camel is it allows you to reconfigure all the routes at runtime.
> This
> > is another reason I liked OSGI support as well as it allows deploying new
> > components without restarting the server (webapp). Of course to scale it
> > would require being able to deploy the configuration across multiple
> > machines which gets fun.
> >
> > Maybe my vision is too big for this project but it's where my head has
> been
> > at.
> >
> > Chris
> >
> >
> > On Wed, Oct 16, 2013 at 7:54 AM, Jason Letourneau
> > <jletourneau80@gmail.com>wrote:
> >
> > > My only real complaint with this approach is that interjecting new
> > > components (processing components) becomes less plug and play without
> > > writing Java code in streams - with the Camel routing approach,
> > > streams Java is a black box and you just change the camel xml.
> > >
> > > The primary rationale in using Camel was gaining all of the
> > > architectural benefits one gains from  enterprise integration patterns
> > > (truth: simplicity is not one and is a tradeoff for flexibility, but
> > > generally is considered a universal truth not one relegated to Camel
> > > and EIP).
> > >
> > > Jason
> > >
> > > On Fri, Oct 11, 2013 at 10:16 AM, Danny Sullivan <
> dsullivan7@hotmail.com>
> > > wrote:
> > > > Hey All,
> > > > I would like to merge the webservice branch with trunk. I think
> making
> > > streams web service based allows for a better focus on the goal of the
> > > application: process activitystrea.ms formatted activity json. While
> > > moving to a webservice model would lose us support for many of the use
> > > cases Chris mentioned (i.e twitter and facebook), I think the downsides
> > > would be mitigated by allowing and encouraging users to write custom
> > > appenders that take in twitter/facebook data, format it to
> > > activitystrea.ms standards, and POST to the Streams Server. These
> > > appenders could even be written in any language as long as the output
> is
> > > activitystrea.ms json. Compared to the trunk branch, I think the web
> > > service design employs the "black box" idiom in a much simpler
> fashion. I
> > > would be happy to hear contradicting thoughts and would encourage a
> > > discussion on this design switch.
> > > > Thanks!
> > > > Danny
> > > >> From: dsullivan7@hotmail.com
> > > >> To: dev@streams.incubator.apache.org
> > > >> Subject: RE: Using Camel for Routing
> > > >> Date: Fri, 4 Oct 2013 16:07:06 -0400
> > > >>
> > > >> I wrote up something quickly so I could demo the functionality as
a
> web
> > > service as opposed to Camel. The new branch is available here:
> > > >>
> https://svn.apache.org/repos/asf/incubator/streams/branches/webservice/
> > > >> note that the new publishing and subscribing urls are:
> > > >>
> > >
> http://localhost:8080/streams-web/app/publisherRegisterhttp://localhost:8080/streams-web/app/subscriberRegister
> > > >> I used a Spring Web Service only because I was more familiar with
> it, I
> > > would like to research other alternatives (CXF) if it's clear that a
> web
> > > service is a realistic alternative. The main changes to the
> application are
> > > the web controller located here:
> > > >>
> > >
> https://svn.apache.org/repos/asf/incubator/streams/branches/webservice/streams-web/src/main/java/org/apache/streams/mvc/controller/StreamsWebController.java
> > > >> Which handles the input and output of the application, which were
> > > handled by the camel context before. The other main change is the use
> of
> > > the streams-components module located here:
> > > >>
> > >
> https://svn.apache.org/repos/asf/incubator/streams/branches/webservice/streams-components/
> > > >> This module contains service classes that return the correct output
> on
> > > each request sent to the controller. I reused the Publisher and
> Subscriber
> > > objects as well as the activityConsumerWarehouse and
> > > activitySubscriberWarehouse, I think they would be moved to the
> > > streams-components package if we want to go this route.
> > > >> Overall, I think the flow through the application is much more
> > > understandable. Previously, a developer would have to follow an
> activity
> > > through the camelContext.xml as well as through dynamically created
> routes
> > > verses the web service which provides clear entry and exit points to
> the
> > > application. I'd really like to hear feedback on this one, I really
> don't
> > > want to lose any efficiencies that we could be from
> > > messaging/osgi/camel/activemq.
> > > >> Thanks,
> > > >> Danny
> > > >>
> > > >> > From: dsullivan7@hotmail.com
> > > >> > To: dev@streams.incubator.apache.org
> > > >> > Subject: RE: Using Camel for Routing
> > > >> > Date: Thu, 3 Oct 2013 20:17:13 -0400
> > > >> >
> > > >> > Thanks for the update Jason. I'll keep the Spring MVC app in
a
> > > separate branch (hopefully it won't take too long) and then perhaps we
> can
> > > reassess having both component and web container deployment? I think
> it's a
> > > good idea to keep both options in mind especially because this
> technology
> > > is relatively young.
> > > >> > Thanks!
> > > >> > Danny
> > > >> >
> > > >> > > Date: Thu, 3 Oct 2013 19:50:59 -0400
> > > >> > > Subject: Re: Using Camel for Routing
> > > >> > > From: jletourneau80@gmail.com
> > > >> > > To: dev@streams.incubator.apache.org
> > > >> > >
> > > >> > > Because of the osgi container support in addition to deployment
> as
> > > a web
> > > >> > > app - when last we visited this topic - I think the majority
> was in
> > > favor
> > > >> > > of keeping the component style deployment in addition to
web
> > > container
> > > >> > > deployment
> > > >> > >
> > > >> > > On Thursday, October 3, 2013, Danny Sullivan wrote:
> > > >> > >
> > > >> > > > Hey all,
> > > >> > > > I've been considering taking a stab at moving streams
from
> it's
> > > current
> > > >> > > > configuration with Camel to a Spring Web Service. Concerns
> over
> > > failover
> > > >> > > > and load balancing would be mitigated by the Cassandra
DB
> > > (running on 9160)
> > > >> > > > and Storm (which depends on an instance of Zookeeper
on 2181).
> > > Before I
> > > >> > > > start lancing any windmills I thought it'd be nice
to get
> input
> > > on why the
> > > >> > > > design choice was made to base the application around
Camel?
> > > >> > > > I think that tracking the flow of an activity through
the
> > > application will
> > > >> > > > be much easier if we use Spring MVC. I'll set up a
new branch
> > > using the MVC
> > > >> > > > architecture and I'll keep the community updated as
I go
> along.
> > > >> > > > Danny
> > > >> >
> > > >>
> > > >
> > >
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message