streams-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Geer <>
Subject Re: Using Camel for Routing
Date Fri, 04 Oct 2013 02:45:20 GMT
On Thu, Oct 3, 2013 at 5:35 PM, Danny Sullivan <>wrote:

> Hey Chris,
> Thanks for the response. The way Streams is currently implemented, Camel
> provides both entry and exit endpoints to the application. These are either
> dynamic (when a publisher or subscriber is registered) or defined in the
> camel context. Camel also provides routing from component to component once
> publishers (ActivityConsumers) and subscribers (ActivitySubscribers) are
> created. Because these classes are serialized so that they can be routed in
> Camel, it made sense to me to keep these objects as they are and pass them
> from class to class via method calls (as opposed of using "from" and "to"
> in the camel context).

Ya, I agree the existing architecture has it quirks. I would personally
like to see something a lot more loosely coupled and message based (not
method based). To me, a publish-subscribe (or at least a queue based async
models) fits streams much better than a request/response model of method
calls. I need to get some time to document my idea and get some feedback on

> The reason I chose a Spring Web Service as opposed to any other library is
> because Spring is already in use in the application and it is what is
> familiar to me. With the use of a Spring MVC all of the entry and exit
> points will be in one Web Controller class and the user can follow an
> activity all the way through the application.
I don't think adding Cassandra and Zookeeper will add too much ease of use
> issues to the application. To incorporate storm (and zookeeper), all that's
> needed is a dependency in the pom, there is no added software that is
> needed by the user. Cassandra on the the other hand does require the user
> to download the Cassandra software, so you're correct in that regard. I'd
> really like to embed Cassandra into the application and is one of the tasks
> I'd like to work on.

 Is there a reason that you would recommend CFX as opposed to Spring MVC?
> I'm all for researching alternatives to a Spring MVC if you think it will
> be easier and lighter.

I have several biases toward CXF:
 - It's another Apache project (this is purely my preference to see Apache
projects linked against other Apache projects when possible)
 - It has great OSGI and WebApp support - Last I heard Spring is removing
OSGI support from it's code [1]. You can get OSGI version from their EBR
repository but that is supposed to be shut down sometime next year.
 - CXF is more stand-alone than Spring MVC. It's purely a webservice
framework while Spring MVC sucks in a bunch of UI related dependencies. CXF
does support both SOAP and RESTful webservices however it also allows you
to get distros that just support the one you need.

> What do you mean by access to various services like Twitter and Facebook?
> Are these services that we would not get access to if we switched from
> Camel?

Camel is an integration framework. It's job is to allow people to build
"routes" between various end points. Those endpoints can be web services
(like streams uses today) but they also support many other endpoints [2]
like Twitter and Facebook. Camel would give Streams a level of
configurability that would be hard to get with just Java. Let's say someone
needs to read messages from twitter, process them for certain information
and then write those processed messages to a file (not great use case I
know). In camel the system could just spin up a new route like this.


I know there has been some talk about replacing Camel with Storm but I
don't think it has support for the same number of endpoints, at least yet.
I have seen a couple places where they were used together. I need to learn
more about Storm but I do have some questions about how the vision is to
use it for Streams if anyone can spell that out. I suspect using Storm
requires a very different approach than is being used now anyway (for
example, embedding Storm in a webapp probably isn't the plan I assume).

> Thanks again for the feedback!
> Danny



> > Date: Thu, 3 Oct 2013 17:14:33 -0700
> > Subject: Re: Using Camel for Routing
> > From:
> > To:
> >
> > On Thursday, October 3, 2013, Danny Sullivan wrote:
> >
> > > Hey all,
> > > I've been considering taking a stab at moving streams from it's current
> > > configuration with Camel to a Spring Web Service.
> >
> >
> > Spring MVC != Camel. The driver for camel was an easily composable
> solution
> > that gets you access to various services like Twitter and Facebook. If
> you
> > just want a web service I'd recommend CXF.
> >
> > Concerns over failover and load balancing would be mitigated by the
> > > Cassandra DB (running on 9160) and Storm (which depends on an instance
> of
> > > Zookeeper on 2181).
> >
> > Just my 2 cents, but adding requirements for Cassandra and Zookeeper
> raises
> > the bar on "ease of use" quite a bit. Not saying it might not be the
> right
> > choice, just and observation.
> >
> >
> >
> > > Before I start lancing any windmills I thought it'd be nice to get
> input
> > > on why the design choice was made to base the application around Camel?
> > > I think that tracking the flow of an activity through the application
> will
> > > be much easier if we use Spring MVC. I'll set up a new branch using
> the MVC
> > > architecture and I'll keep the community updated as I go along.
> >
> > What is the primary driver for introducing another major framework like
> > Spring?
> >
> > > Danny

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message