streams-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Geer <ch...@cxtsoftware.com>
Subject Re: [DISCUSS] Switching Streams from Camel deployment to .war deployment
Date Wed, 30 Oct 2013 16:50:38 GMT
Ok, lots of thoughts in-line.

On Wed, Oct 30, 2013 at 8:37 AM, Danny Sullivan <dsullivan7@hotmail.com>wrote:

> My argument is not for the IoC pattern as that can be (and has been)
> implemented alongside Camel. My main argument is that the syntax at the
> entry point is not only familiar but much simpler. This wouldn't be a very
> strong argument if the Camel implementation wasn't much more complicated
> but I feel that it is the case. Also, looking toward the future, if the
> server is restarted, in-routes are lost in Camel.


Yes, just like if you restarted a server your in-memory processing in a web
app would be lost. The way to solve this is to use a storage mechanism to
store state/messages. I believe the current implementation does that with
ActiveMQ which provides both disk and DB storage of messages.


> The way to curb this is to persist the dynamic routes that Camel creates,
> and then on start up pull every one of these routes and recreate a dynamic
> route for each one. Not only is this much easier to implement using the
> Spring web implementation, but it already has been implemented and you can
> try it by checking out the webservice branch, registering a subscriber,
> restarting tomcat, and using the same url you had before. This will allow
> subscribers to hang on to their urls once they register. (the same is true
> for publishers: you can post via the same url after restarting tomcat)
>
> > Date: Wed, 30 Oct 2013 11:00:21 -0400
> > Subject: Re: [DISCUSS] Switching Streams from Camel deployment to .war
> deployment
> > From: jletourneau80@gmail.com
> > To: dev@streams.incubator.apache.org
> >
> > To be fair, while the current implementation is heavily camel-based,
> > all of the interfaces related to Streams functionality are not.  The
> > current model maps to what Matt has outlined in my opinion, though
> > packing names etc. probably don't follow that exact pattern.
>

One of the problems I have with the current implementation (both of them)
are the interfaces. In my opinion, to make this work at scale we need to
forget about Java interfaces and focus on message formats. Sure there needs
to be an API to get data in and data out but in between you can scale a
message based system (Storm or Camel/ActiveMQ) easier than an RPC system
(i.e. Java method calls).

> >
> > With regards to the complexity and different components in the
> > registration process, this was a cut at the abstraction based on the
> > assumption that different implementations may be plugged in and in
> > fact may live on different processor space (ie. a polling publisher vs
> > a push publisher may be instantiated on different servers but the
> > registration URL is staticly defined).
> >
> > Is the main argument I am seeing for Spring  the familiarity of its
> > IoC pattern implementation and syntax at the entry point?
> >
> > On Wed, Oct 30, 2013 at 10:53 AM, Danny Sullivan <dsullivan7@hotmail.com>
> wrote:
> > > Could you clarify whether the same entry points would exist for the
> camel implementation of the core (implementing the "process" method/ using
> a DynammicRouteBuilder) or would the webservice be the sole entry point to
> Streams and after it enters would it hand it off to Camel? And what would
> be the entry point for the Storm implementation?
> > > -Danny
> > >
> > >> Date: Wed, 30 Oct 2013 10:13:04 -0400
> > >> Subject: Re: [DISCUSS] Switching Streams from Camel deployment to
> .war deployment
> > >> From: m.ben.franklin@gmail.com
> > >> To: dev@streams.incubator.apache.org
> > >>
> > >> On Tue, Oct 29, 2013 at 2:55 PM, Danny Sullivan <
> dsullivan7@hotmail.com>wrote:
> > >>
> > >> > My case for switching from OSGi is for simplicity in design. To
> follow the
> > >> > path of an activity through Streams in the webservice, there is one
> main
> > >> > things the developer needs to understand:
> > >> > The @RequestMapping annotation specifies the HTTP entry point
> > >> > There are 4 @RequestMapping annotations that correspond to each of
> the 4
> > >> > ways a user enters the application: registering a publisher,
> registering a
> > >> > subscriber, publishing activity, and getting an activity stream.
> Where
> > >> > these are located in the source code can be found by searching for
> the
> > >> > paths specified in the documentation (search for
> "/publisherRegister",
> > >> > "/subscriberRegister", "/publishActivity", "/getActivity" which
> will all
> > >> > lead you to StreamsWebController.java). From the methods that
> process
> > >> > requests, the flow through the application is through methods which
> can be
> > >> > understood by most Java programmers.
> > >> > The flow of activities through the current trunk branch is
> understood as
> > >> > follows:
> > >> > The string "/publisher/register" (the entry point to register a
> publisher
> > >> > specified in the documentation) is the value of the
> > >> > consumer.registrationEndpoint property defined in
> streams.propertiesThe
> > >> > camelContext.xml specifies an endpoint with the id
> > >> > consumerRegistrationEndpoint to have a uri equal to the propertyThe
> > >> > consumerRegistrationEndpoint routes from uri direct:publisher
> register with
> > >> > the bean activityRegistrationProcessor nested in between the
> routeThe
> > >> > streams-eip-applicationContext contains a bean with the id
> > >> > activityRegistrationProcessor created for the class
> > >> > ActivityPublisherRegistrationProcessorThe exchange will enter the
> "process"
> > >> > method of the class ActivityPublisherRegisitrationProcessor and
> that this
> > >> > is because this class implements the "Processor" interface provided
> by
> > >> > CamelThe direct:add-publisher-route takes the exchange output from
> the
> > >> > "process" method and routes it to the activityRegistrationProcessor
> > >> > "register" method. The bean activityRegistrationProcessor is
> defined in the
> > >> > streams-eip-osgi-component-import.xmlThe output from this method is
> then
> > >> > sent to the "createNewRouteForConsumer" method of
> activityConsumerRouter.
> > >> > This method creates a new route for the newly registered publisher
> using
> > >> > the private static final class DynamicConsumerRouteBuilder which is
> > >> > required to extend RouteBuilder which is provided by Camel. This
> > >> > DynamicConsumerRouteBuilder contains several methods:
> > >> > getConsumerReceiveMethod() (which corresponds to @Value
> > >> > ${consumer.receiveMethod} which corresponds to "receive"),
> > >> > getConsumerSplitMethod() (@Value ${consumer.splitMethod},"split"),
> and
> > >> > getConsumerActivityQUri() (@Value ${consumerActivityQUri},
> > >> > "direct:activityQ"). This is different than the camelContext.xml in
> that
> > >> > the route is being created programmatically. What this is doing is
> routing
> > >> > input from the inroute url (which Camel does automatically through
> the
> > >> > configure method which is required to be overridden), to the
> "receive"
> > >> > method of ActivityConsumer, then to the "split" method of
> ActivityConsumer,
> > >> > and then to the "direct:activityQ" which if you look back in the
> > >> > camelContext.xml routes to the "activemq:queue:activities" which
> then
> > >> > routes to "receiveExchange"
> > >> > This is the process to register a publisher. The process for
> registering a
> > >> > subscriber is relatively the same though it involves separate
> classes with
> > >> > their own private static final RouteBuilder class. From my
> perspective, the
> > >> > two most difficult things with setting this project up were
> understanding
> > >> > that the "process" method of the class that implements "Processor"
> is the
> > >> > entry point and the DynamicConsumerRouteBuilder creates the second
> entry
> > >> > point (The 5th and last points). This made the project very, VERY
> hard to
> > >> > understand.
> > >> > In addition to simplicity of design, the mvn clean install of the
> web
> > >> > service project is much faster and small scale activity publishing
> is also
> > >> > faster (see my email about load testing). These are minor points
> though as
> > >> > compilation has no effect on deployment. OSGi does add the benefit
> of
> > >> > modularized programming which is valuable, though I think the added
> > >> > complexity of Camel merits moving the project away from this
> paradigm.
>

Camel != OSGI. Camel can run just fine inside a WAR and you can have an
OSGI based application without Camel. OSGI is a deployment model where you
have well defined interfaces between components and isolation. It's more of
a competitor with Spring although you can run Spring components in an OSGI
server. In fact Spring DM is now Eclipse Gemini which is an OSGI Blueprint
implementation.


> > >> >
> > >>
> > >> I agree that the project is pretty difficult to understand ATM.  I
> think
> > >> what we need to do is think about what the responsibilities of the
> code are
> > >> and allow for different implementations that are not so tightly
> coupled as
> > >> they are now.  For instance, having worked with Storm to ingest
> millions of
> > >> activities a day, I personally would like to see streams be
> responsible for
> > >> defining an over-arching orchestration model that can be implemented
> within
> > >> a single war or on top of a distributed system.  This would look
> something
> > >> like the follows
>
> >>
> > >> |__ Streams-Core  (Base classes, interfaces, utilities, extensions,
> etc)
> > >> |
> > >> |__ Streams-Storm (Storm implementation of the core)
> > >> |
> > >> |__ Streams-Camel (Camel implementation of the core)
> > >> |
> > >> |__ Streams-WS (Web service implementation)
>

Matt, I understand what you are trying to do here but again, big picture
I'm sure sure it's this clean cut. Maybe it is for simple implementations
but I would think larger system would require a combination of things like
Storm and Camel working together.


> > >>
> > >>
> > >> > Danny
> > >> >
> > >> >
> > >> > > From: dsullivan7@hotmail.com
> > >> > > To: dev@streams.incubator.apache.org
> > >> > > Subject: [DISCUSS] Switching Streams from Camel deployment to
.war
> > >> > deployment
> > >> > > Date: Tue, 29 Oct 2013 11:07:39 -0400
> > >> > >
> > >> > > The discussion thread for switching Streams from
> Camel/osgi/Servicemix
> > >> > to a single .war deployment
> > >> >
> > >> >
> > >
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message