streams-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Danny Sullivan <dsulliv...@hotmail.com>
Subject RE: [DISCUSS] Switching Streams from Camel deployment to .war deployment
Date Wed, 30 Oct 2013 18:48:52 GMT
I don't believe that urls are able to be reused when restarting the application, at least in
my experience, though I could be wrong. Do you have steps to reproduce starting and stoping
a server while maintaining the same url? Also note, that stopping the .war would only lose
activities/publishers/subscriptions currently being processed, not the ones successfully stored
in the database.

I'm not sure I follow you on focusing on message formats as opposed to interfaces. Could you
explain a little more?

I agree that it will not involve something such as a Camel implementation, a Storm implementation,
and a Web implementation. I originally imagined the Spring controller acting as an entry point
to Storm, but perhaps that remains to be seen.
> Date: Wed, 30 Oct 2013 09:50:38 -0700
> Subject: Re: [DISCUSS] Switching Streams from Camel deployment to .war deployment
> From: chris@cxtsoftware.com
> To: dev@streams.incubator.apache.org
> 
> Ok, lots of thoughts in-line.
> 
> On Wed, Oct 30, 2013 at 8:37 AM, Danny Sullivan <dsullivan7@hotmail.com>wrote:
> 
> > My argument is not for the IoC pattern as that can be (and has been)
> > implemented alongside Camel. My main argument is that the syntax at the
> > entry point is not only familiar but much simpler. This wouldn't be a very
> > strong argument if the Camel implementation wasn't much more complicated
> > but I feel that it is the case. Also, looking toward the future, if the
> > server is restarted, in-routes are lost in Camel.
> 
> 
> Yes, just like if you restarted a server your in-memory processing in a web
> app would be lost. The way to solve this is to use a storage mechanism to
> store state/messages. I believe the current implementation does that with
> ActiveMQ which provides both disk and DB storage of messages.
> 
> 
> > The way to curb this is to persist the dynamic routes that Camel creates,
> > and then on start up pull every one of these routes and recreate a dynamic
> > route for each one. Not only is this much easier to implement using the
> > Spring web implementation, but it already has been implemented and you can
> > try it by checking out the webservice branch, registering a subscriber,
> > restarting tomcat, and using the same url you had before. This will allow
> > subscribers to hang on to their urls once they register. (the same is true
> > for publishers: you can post via the same url after restarting tomcat)
> >
> > > Date: Wed, 30 Oct 2013 11:00:21 -0400
> > > Subject: Re: [DISCUSS] Switching Streams from Camel deployment to .war
> > deployment
> > > From: jletourneau80@gmail.com
> > > To: dev@streams.incubator.apache.org
> > >
> > > To be fair, while the current implementation is heavily camel-based,
> > > all of the interfaces related to Streams functionality are not.  The
> > > current model maps to what Matt has outlined in my opinion, though
> > > packing names etc. probably don't follow that exact pattern.
> >
> 
> One of the problems I have with the current implementation (both of them)
> are the interfaces. In my opinion, to make this work at scale we need to
> forget about Java interfaces and focus on message formats. Sure there needs
> to be an API to get data in and data out but in between you can scale a
> message based system (Storm or Camel/ActiveMQ) easier than an RPC system
> (i.e. Java method calls).
> 
> > >
> > > With regards to the complexity and different components in the
> > > registration process, this was a cut at the abstraction based on the
> > > assumption that different implementations may be plugged in and in
> > > fact may live on different processor space (ie. a polling publisher vs
> > > a push publisher may be instantiated on different servers but the
> > > registration URL is staticly defined).
> > >
> > > Is the main argument I am seeing for Spring  the familiarity of its
> > > IoC pattern implementation and syntax at the entry point?
> > >
> > > On Wed, Oct 30, 2013 at 10:53 AM, Danny Sullivan <dsullivan7@hotmail.com>
> > wrote:
> > > > Could you clarify whether the same entry points would exist for the
> > camel implementation of the core (implementing the "process" method/ using
> > a DynammicRouteBuilder) or would the webservice be the sole entry point to
> > Streams and after it enters would it hand it off to Camel? And what would
> > be the entry point for the Storm implementation?
> > > > -Danny
> > > >
> > > >> Date: Wed, 30 Oct 2013 10:13:04 -0400
> > > >> Subject: Re: [DISCUSS] Switching Streams from Camel deployment to
> > .war deployment
> > > >> From: m.ben.franklin@gmail.com
> > > >> To: dev@streams.incubator.apache.org
> > > >>
> > > >> On Tue, Oct 29, 2013 at 2:55 PM, Danny Sullivan <
> > dsullivan7@hotmail.com>wrote:
> > > >>
> > > >> > My case for switching from OSGi is for simplicity in design.
To
> > follow the
> > > >> > path of an activity through Streams in the webservice, there
is one
> > main
> > > >> > things the developer needs to understand:
> > > >> > The @RequestMapping annotation specifies the HTTP entry point
> > > >> > There are 4 @RequestMapping annotations that correspond to each
of
> > the 4
> > > >> > ways a user enters the application: registering a publisher,
> > registering a
> > > >> > subscriber, publishing activity, and getting an activity stream.
> > Where
> > > >> > these are located in the source code can be found by searching
for
> > the
> > > >> > paths specified in the documentation (search for
> > "/publisherRegister",
> > > >> > "/subscriberRegister", "/publishActivity", "/getActivity" which
> > will all
> > > >> > lead you to StreamsWebController.java). From the methods that
> > process
> > > >> > requests, the flow through the application is through methods
which
> > can be
> > > >> > understood by most Java programmers.
> > > >> > The flow of activities through the current trunk branch is
> > understood as
> > > >> > follows:
> > > >> > The string "/publisher/register" (the entry point to register
a
> > publisher
> > > >> > specified in the documentation) is the value of the
> > > >> > consumer.registrationEndpoint property defined in
> > streams.propertiesThe
> > > >> > camelContext.xml specifies an endpoint with the id
> > > >> > consumerRegistrationEndpoint to have a uri equal to the propertyThe
> > > >> > consumerRegistrationEndpoint routes from uri direct:publisher
> > register with
> > > >> > the bean activityRegistrationProcessor nested in between the
> > routeThe
> > > >> > streams-eip-applicationContext contains a bean with the id
> > > >> > activityRegistrationProcessor created for the class
> > > >> > ActivityPublisherRegistrationProcessorThe exchange will enter
the
> > "process"
> > > >> > method of the class ActivityPublisherRegisitrationProcessor and
> > that this
> > > >> > is because this class implements the "Processor" interface provided
> > by
> > > >> > CamelThe direct:add-publisher-route takes the exchange output
from
> > the
> > > >> > "process" method and routes it to the activityRegistrationProcessor
> > > >> > "register" method. The bean activityRegistrationProcessor is
> > defined in the
> > > >> > streams-eip-osgi-component-import.xmlThe output from this method
is
> > then
> > > >> > sent to the "createNewRouteForConsumer" method of
> > activityConsumerRouter.
> > > >> > This method creates a new route for the newly registered publisher
> > using
> > > >> > the private static final class DynamicConsumerRouteBuilder which
is
> > > >> > required to extend RouteBuilder which is provided by Camel. This
> > > >> > DynamicConsumerRouteBuilder contains several methods:
> > > >> > getConsumerReceiveMethod() (which corresponds to @Value
> > > >> > ${consumer.receiveMethod} which corresponds to "receive"),
> > > >> > getConsumerSplitMethod() (@Value ${consumer.splitMethod},"split"),
> > and
> > > >> > getConsumerActivityQUri() (@Value ${consumerActivityQUri},
> > > >> > "direct:activityQ"). This is different than the camelContext.xml
in
> > that
> > > >> > the route is being created programmatically. What this is doing
is
> > routing
> > > >> > input from the inroute url (which Camel does automatically through
> > the
> > > >> > configure method which is required to be overridden), to the
> > "receive"
> > > >> > method of ActivityConsumer, then to the "split" method of
> > ActivityConsumer,
> > > >> > and then to the "direct:activityQ" which if you look back in
the
> > > >> > camelContext.xml routes to the "activemq:queue:activities" which
> > then
> > > >> > routes to "receiveExchange"
> > > >> > This is the process to register a publisher. The process for
> > registering a
> > > >> > subscriber is relatively the same though it involves separate
> > classes with
> > > >> > their own private static final RouteBuilder class. From my
> > perspective, the
> > > >> > two most difficult things with setting this project up were
> > understanding
> > > >> > that the "process" method of the class that implements "Processor"
> > is the
> > > >> > entry point and the DynamicConsumerRouteBuilder creates the second
> > entry
> > > >> > point (The 5th and last points). This made the project very,
VERY
> > hard to
> > > >> > understand.
> > > >> > In addition to simplicity of design, the mvn clean install of
the
> > web
> > > >> > service project is much faster and small scale activity publishing
> > is also
> > > >> > faster (see my email about load testing). These are minor points
> > though as
> > > >> > compilation has no effect on deployment. OSGi does add the benefit
> > of
> > > >> > modularized programming which is valuable, though I think the
added
> > > >> > complexity of Camel merits moving the project away from this
> > paradigm.
> >
> 
> Camel != OSGI. Camel can run just fine inside a WAR and you can have an
> OSGI based application without Camel. OSGI is a deployment model where you
> have well defined interfaces between components and isolation. It's more of
> a competitor with Spring although you can run Spring components in an OSGI
> server. In fact Spring DM is now Eclipse Gemini which is an OSGI Blueprint
> implementation.
> 
> 
> > > >> >
> > > >>
> > > >> I agree that the project is pretty difficult to understand ATM.  I
> > think
> > > >> what we need to do is think about what the responsibilities of the
> > code are
> > > >> and allow for different implementations that are not so tightly
> > coupled as
> > > >> they are now.  For instance, having worked with Storm to ingest
> > millions of
> > > >> activities a day, I personally would like to see streams be
> > responsible for
> > > >> defining an over-arching orchestration model that can be implemented
> > within
> > > >> a single war or on top of a distributed system.  This would look
> > something
> > > >> like the follows
> >
> > >>
> > > >> |__ Streams-Core  (Base classes, interfaces, utilities, extensions,
> > etc)
> > > >> |
> > > >> |__ Streams-Storm (Storm implementation of the core)
> > > >> |
> > > >> |__ Streams-Camel (Camel implementation of the core)
> > > >> |
> > > >> |__ Streams-WS (Web service implementation)
> >
> 
> Matt, I understand what you are trying to do here but again, big picture
> I'm sure sure it's this clean cut. Maybe it is for simple implementations
> but I would think larger system would require a combination of things like
> Storm and Camel working together.
> 
> 
> > > >>
> > > >>
> > > >> > Danny
> > > >> >
> > > >> >
> > > >> > > From: dsullivan7@hotmail.com
> > > >> > > To: dev@streams.incubator.apache.org
> > > >> > > Subject: [DISCUSS] Switching Streams from Camel deployment
to .war
> > > >> > deployment
> > > >> > > Date: Tue, 29 Oct 2013 11:07:39 -0400
> > > >> > >
> > > >> > > The discussion thread for switching Streams from
> > Camel/osgi/Servicemix
> > > >> > to a single .war deployment
> > > >> >
> > > >> >
> > > >
> >
> >
 		 	   		  
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message