streams-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Danny Sullivan <dsulliv...@hotmail.com>
Subject RE: [DISCUSS] Switching Streams from Camel deployment to .war deployment
Date Fri, 01 Nov 2013 18:06:57 GMT
Excellent, thanks very much for the info. This does help me understand why you would want to use EIP for development of certain systems. Because I understand the value of this design I think it would be a good idea to keep the current EIP implementation as the trunk branch and retract my request to merge the webservice branch with trunk at this time. However, I think it would also still be valuable to draw out the webservice branch and explore Streams as an "application" rather than a "system", at least until there is a crystal clear vision of what the direction should be.

I think both of these application/systems will move in the same direction and ultimately fulfill the initial design goal of a "lightweight (yet scalable) server for ActivityStreams". To keep this goal in mind, I'd like to keep the discussion going by posing a scenario. Consider that Streams is completely developed. What are the answers to these questions in this scenario:

"What does Streams do?"

"How do I enable my application to support Streams?" 


Thanks very much for all the feedback, this has been a very good discussion!
Danny

> Date: Fri, 1 Nov 2013 12:53:28 -0400
> Subject: Re: [DISCUSS] Switching Streams from Camel deployment to .war deployment
> From: jletourneau80@gmail.com
> To: dev@streams.incubator.apache.org
> 
> Nice Chris - well put.  Streams isn't an application (or shouldn't be
> thought of as one) it is a system that potentially is part of a system
> of systems.
> 
> To punctuate your example, I also like to recall the classic EIP
> banking/loan example [3]
> 
> [3] http://www.eaipatterns.com/ComposedMessagingExample.html
> 
> On Fri, Nov 1, 2013 at 12:48 PM, Chris Geer <chris@cxtsoftware.com> wrote:
> > On Fri, Nov 1, 2013 at 6:56 AM, Danny Sullivan <dsullivan7@hotmail.com>wrote:
> >
> >> Perhaps it would be best for me to take a step back and ask what problem
> >> are we trying to solve by using Camel? My understanding is that it is to
> >> allow distribution of Streams to multiple servers. It would be helpful to
> >> me to see another open source application that has this deployment
> >> structure (it doesn't have to use Camel, just an application that has
> >> multiple jars communicating with one another). Does anyone have any
> >> suggestions of software for me to look at?
> >>
> >
> > Danny, I wish I could just show you my system we are building. Honestly I
> > think you are looking at this the wrong way. You are looking at it from a
> > software development perspective (jars, protocols) and not from a
> > business/system perspective (messages, components). So let me use an
> > example.
> >
> > Let's say I have a business that takes order for widgets. I want the order
> > taking to be streamlined for the user. To make that happen my order entry
> > system does some basic checks when the user is placing the order and if
> > those pass it saves the order to the database. To kick off the other
> > processing, whenever I save an order to the database I sent a message to a
> > Topic (pub/sub) saying either an order was created or an order was updated.
> > Now at this point there is a lot more work I have to do to get the order
> > shipped to the customer but it's an async action to the order taking. At
> > this point the order has been placed and the customer can go about their
> > business.
> >
> > I also have a component I wrote that listens to the Topic for "Order
> > Created" messages. It takes each message, retrieves the order does some
> > inventory processing to get the order ready to ship. At this point the
> > order is complete and shipped to the customer.
> >
> > Now, in this simple example you could argue that the order entry system
> > could just call the order processing component directly but that would
> > preclude some extensibility in the future.
> >
> > A few months down the road I get a requirement to be able to notify the
> > sales team whenever an order is placed by a certain set of customers. In a
> > traditional model I'd have to modify the order entry component and put in
> > logic to handle that. In this architecture all I have to do is build a
> > different component that listens to that same Topic for order by certain
> > customers. When it gets a notification it can then do the logic to send the
> > message. The beauty of this is the order entry system doesn't' have to be
> > modified and there is no risk of inserting a bug into that core process.
> >
> > So you ask, where does camel come in here. It could be used in lots of
> > places but the natural fit is for the shipment processing component to be a
> > camel route that listens to the Topic and calls a java object when messages
> > come in. Also for the notification components, the whole thing could
> > probably be a camel route (0 Java code).
> >
> > This is a pretty simple example and there are lots of other benefits I
> > didn't talk about like redundancy and deployment flexibility.
> >
> > Does that help?
> >
> >>
> >> > Date: Thu, 31 Oct 2013 20:23:26 -0400
> >> > Subject: Re: [DISCUSS] Switching Streams from Camel deployment to .war
> >> deployment
> >> > From: jletourneau80@gmail.com
> >> > To: dev@streams.incubator.apache.org
> >> >
> >> > So Danny in case it was lost in the ensuing transport discussion,
> >> > Camel abstracts the transport, so you don't have to care if its http
> >> > or whatever, only about the message format.  This is helpful if you
> >> > care about plugging in components that DO care about the transport of
> >> > course.  To answer your other questions, the downside to having the
> >> > components all work via http (and not using messaging/Camel) is that
> >> > everything immediately becomes point to point, which isn't a
> >> > performance issue in its own right but limits scaling to more than one
> >> > component being at the receiving end/fulfillment for instance as we've
> >> > been discussing as being optimal.
> >> >
> >> > On Thu, Oct 31, 2013 at 5:38 PM, Matt Franklin <m.ben.franklin@gmail.com>
> >> wrote:
> >> > > On Thu, Oct 31, 2013 at 5:34 PM, Chris Geer <chris@cxtsoftware.com>
> >> wrote:
> >> > >
> >> > >> On Thu, Oct 31, 2013 at 2:32 PM, Matt Franklin <
> >> m.ben.franklin@gmail.com
> >> > >> >wrote:
> >> > >>
> >> > >> > On Thu, Oct 31, 2013 at 5:28 PM, Chris Geer <chris@cxtsoftware.com>
> >> > >> wrote:
> >> > >> >
> >> > >> > > On Thu, Oct 31, 2013 at 2:23 PM, Matt Franklin <
> >> > >> m.ben.franklin@gmail.com
> >> > >> > > >wrote:
> >> > >> > >
> >> > >> > > > On Thu, Oct 31, 2013 at 5:04 PM, Danny Sullivan <
> >> > >> > dsullivan7@hotmail.com
> >> > >> > > > >wrote:
> >> > >> > > >
> >> > >> > > > > I'm not quite following this, so I apologize. What I'm trying
> >> to do
> >> > >> > is
> >> > >> > > > > programmatically make a request to a jar running on a
> >> separate jvm
> >> > >> > and
> >> > >> > > > get
> >> > >> > > > > the response from that call all within the same method.
> >> Similar to
> >> > >> > this
> >> > >> > > > > http request:
> >> > >> > > > >
> >> > >> > > > > HttpGet httpget = new HttpGet();
> >> > >> > > > >
> >> > >> > > > > httpget.setURI(new URI("www.streams-persistence.com"));
> >> > >> > > > >
> >> > >> > > > > CloseableHttpResponse response = httpclient.execute(httpget);
> >> > >> > > > >
> >> > >> > > > > //do stuff with the response...
> >> > >> > > > >
> >> > >> > > > >
> >> > >> > > > >
> >> > >> > > > >
> >> > >> > > > > I imagine that this would translate to something in Camel
> >> similar
> >> > >> to
> >> > >> > > > this:
> >> > >> > > > >
> >> > >> > > > >
> >> > >> > > > > <route>
> >> > >> > > > >
> >> > >> > > > > <from uri="bean:subscriberService?method=getAllSubscribers"/>
> >> > >> > > > >
> >> > >> > > > > <inOut
> >> > >> > > > >
> >> > >> > > >
> >> > >> > >
> >> > >> >
> >> > >>
> >> uri="addressToStreamsPersistenceJar?method=selectAllSubscribersFromDatabase"/>
> >> > >> > > > >
> >> > >> > > > > </route>
> >> > >> > > > >
> >> > >> > > >
> >> > >> > > > Camel uses AMQP as a messaging system so that you don't have to
> >> > >> create
> >> > >> > > HTTP
> >> > >> > > > requests between sources, though it is possible to do so.  In
> >> Storm,
> >> > >> we
> >> > >> > > > usually try to use a buffer system like Kafka to do the same.
> >> > >> > > >
> >> > >> > >
> >> > >> > > Camel can use AMQP through ActiveMQ.
> >> > >> > >
> >> > >> > >
> >> > >> > Can being the operative word.  I haven't done much with Camel.  What
> >> > >> other
> >> > >> > transports are available.
> >> > >> >
> >> > >> > OpenWire (native ActiveMQ)
> >> > >> AMQP
> >> > >> STOMP
> >> > >> HTTP (can be a server or client)
> >> > >> Mail
> >> > >> Amazon SQS
> >> > >> XMPP
> >> > >> ....hundred or so more
> >> > >>
> >> > >> http://camel.apache.org/components.html
> >> > >
> >> > >
> >> > >
> >> > > Nice.  I was expecting RTFM, so thanks for catering to my laziness.
> >> > >
> >> > >
> >> > >>
> >> > >>
> >> > >> >
> >> > >> > > >
> >> > >> > > >
> >> > >> > > > >
> >> > >> > > > >
> >> > >> > > > > But it is unclear what the actual implementation would be.
> >> > >> > > > >
> >> > >> > > > >
> >> > >> > > > > This actually brings me to another suggestion. Would there be
> >> a big
> >> > >> > > > > performance impact to have communication between the software
> >> > >> > > components
> >> > >> > > > > occur between http? Say the 5 software components I outlined
> >> > >> earlier
> >> > >> > > were
> >> > >> > > > > packaged as 5 separate wars. These wars could communicate
> >> with each
> >> > >> > > other
> >> > >> > > > > via get a post requests. This sounds unconventional offhand
> >> so I'd
> >> > >> > like
> >> > >> > > > to
> >> > >> > > > > hear some thoughts on it.
> >> > >> > > > >
> >> > >> > > >
> >> > >> > > > It is most certainly possible, but IMO probably not the best
> >> option
> >> > >> for
> >> > >> > > > success.  Protocols like PubSubHubbub use HTTP for a message
> >> > >> transport.
> >> > >> > > >
> >> > >> > > >
> >> > >> > > > > -Danny
> >> > >> > > > >
> >> > >> > > > > > Date: Thu, 31 Oct 2013 13:28:41 -0700
> >> > >> > > > > > Subject: Re: [DISCUSS] Switching Streams from Camel
> >> deployment to
> >> > >> > > .war
> >> > >> > > > > deployment
> >> > >> > > > > > From: chris@cxtsoftware.com
> >> > >> > > > > > To: dev@streams.incubator.apache.org
> >> > >> > > > > >
> >> > >> > > > > > Or Content Enricher [2]
> >> > >> > > > > >
> >> > >> > > > > > [2] http://camel.apache.org/content-enricher.html
> >> > >> > > > > >
> >> > >> > > > > >
> >> > >> > > > > > On Thu, Oct 31, 2013 at 12:37 PM, Jason Letourneau
> >> > >> > > > > > <jletourneau80@gmail.com>wrote:
> >> > >> > > > > >
> >> > >> > > > > > > check out the link here[1]
> >> > >> > > > > > >
> >> > >> > > > > > >
> >> > >> > > > > > > [1]http://camel.apache.org/request-reply.html
> >> > >> > > > > > >
> >> > >> > > > > > > On Thu, Oct 31, 2013 at 2:32 PM, Danny Sullivan <
> >> > >> > > > > dsullivan7@hotmail.com>
> >> > >> > > > > > > wrote:
> >> > >> > > > > > > > I have a quick Camel question that I arrived at in the
> >> > >> > > > > implementation of
> >> > >> > > > > > > these new components:
> >> > >> > > > > > > >
> >> > >> > > > > > > > Lets say I have a method in streams-activity.jar that
> >> needs
> >> > >> all
> >> > >> > > > > > > subscribers in the database. This would require a call to
> >> the
> >> > >> > > > > > > streams-persistence.jar. So far, I've seen camel used
> >> mostly
> >> > >> for
> >> > >> > > > > passing
> >> > >> > > > > > > data through the application, but not for making a single
> >> > >> > > > > request-reponse
> >> > >> > > > > > > from within a method. How can I use Camel to get a list
> >> of all
> >> > >> > > > > subscribers
> >> > >> > > > > > > in the streams-persistence.jar from the
> >> streams-activity.jar?
> >> > >> > > > > > > >
> >> > >> > > > > > > >> From: dsullivan7@hotmail.com
> >> > >> > > > > > > >> To: dev@streams.incubator.apache.org
> >> > >> > > > > > > >> Subject: RE: [DISCUSS] Switching Streams from Camel
> >> > >> deployment
> >> > >> > > to
> >> > >> > > > > .war
> >> > >> > > > > > > deployment
> >> > >> > > > > > > >> Date: Thu, 31 Oct 2013 08:40:44 -0400
> >> > >> > > > > > > >>
> >> > >> > > > > > > >> Excellent, I'll write up something as a proof of
> >> concept and
> >> > >> > we
> >> > >> > > > can
> >> > >> > > > > > > discuss further to make sure everything is vanilla.
> >> > >> > > > > > > >>
> >> > >> > > > > > > >> > Date: Wed, 30 Oct 2013 17:01:28 -0400
> >> > >> > > > > > > >> > Subject: Re: [DISCUSS] Switching Streams from Camel
> >> > >> > deployment
> >> > >> > > > to
> >> > >> > > > > > > .war deployment
> >> > >> > > > > > > >> > From: jletourneau80@gmail.com
> >> > >> > > > > > > >> > To: dev@streams.incubator.apache.org
> >> > >> > > > > > > >> >
> >> > >> > > > > > > >> > That sounds pretty promising to me.
> >> > >> > > > > > > >> >
> >> > >> > > > > > > >> > On Wed, Oct 30, 2013 at 2:38 PM, Danny Sullivan <
> >> > >> > > > > > > dsullivan7@hotmail.com> wrote:
> >> > >> > > > > > > >> > > Thanks for the feedback. You have an interesting
> >> point
> >> > >> > about
> >> > >> > > > the
> >> > >> > > > > > > url linking to a separate processing space. Let me tie my
> >> > >> answer
> >> > >> > > into
> >> > >> > > > > your
> >> > >> > > > > > > last question about "advocating for the simplicity at
> >> > >> > registration
> >> > >> > > to
> >> > >> > > > > give
> >> > >> > > > > > > up flexibility at registration, but retaining the inner
> >> "guts"
> >> > >> of
> >> > >> > > > > > > EIP/messaging". Consider a new architecture:
> >> > >> > > > > > > >> > >
> >> > >> > > > > > > >> > > streams-web.war: single entry point to
> >> application, but
> >> > >> > > > > functions
> >> > >> > > > > > > ONLY as an entry point. From here Camel routes the
> >> incoming
> >> > >> > > requests
> >> > >> > > > > to 4
> >> > >> > > > > > > separate jarssubscriber-registration.jar: subscriber
> >> > >> registration
> >> > >> > > > > > > publisher-registration.jar: publisher
> >> registrationactivity.jar:
> >> > >> > > > returns
> >> > >> > > > > > > activity (also contains subscriber warehouse and storm
> >> activity
> >> > >> > > > > > > aggregator)publish.jar: publishes
> >> > >> activitystreams-cassandra.jar:
> >> > >> > > the
> >> > >> > > > > above
> >> > >> > > > > > > 4 jars would all have a hook into this jar which would
> >> function
> >> > >> > as
> >> > >> > > a
> >> > >> > > > > hook
> >> > >> > > > > > > onto the database. Each jar would have camel route output
> >> to
> >> > >> this
> >> > >> > > > jar.
> >> > >> > > > > > > >> > >
> >> > >> > > > > > > >> > > In this implementation, Camel would no longer be
> >> the
> >> > >> entry
> >> > >> > > and
> >> > >> > > > > exit
> >> > >> > > > > > > point of a client to the application, but would handle the
> >> > >> > > > > communication
> >> > >> > > > > > > between components. The flow of activity through the
> >> > >> application
> >> > >> > > > would
> >> > >> > > > > be
> >> > >> > > > > > > method based in each jar. This would allow deployment on
> >> up to
> >> > >> 6
> >> > >> > > > > different
> >> > >> > > > > > > process spaces. However, this does not address that there
> >> is a
> >> > >> > > single
> >> > >> > > > > > > server entry point, but I'm not sure if it was a concern
> >> in the
> >> > >> > > first
> >> > >> > > > > place.
> >> > >> > > > > > > >> > >
> >> > >> > > > > > > >> > > My argument, at its basis, is that we should move
> >> away
> >> > >> > from
> >> > >> > > > > using
> >> > >> > > > > > > Camel as the entry point to the application. I would be
> >> happy
> >> > >> to
> >> > >> > > > > maintain
> >> > >> > > > > > > messaging between components.
> >> > >> > > > > > > >> > >
> >> > >> > > > > > > >> > >> Date: Wed, 30 Oct 2013 12:32:55 -0400
> >> > >> > > > > > > >> > >> Subject: Re: [DISCUSS] Switching Streams from
> >> Camel
> >> > >> > > > deployment
> >> > >> > > > > to
> >> > >> > > > > > > .war deployment
> >> > >> > > > > > > >> > >> From: jletourneau80@gmail.com
> >> > >> > > > > > > >> > >> To: dev@streams.incubator.apache.org
> >> > >> > > > > > > >> > >>
> >> > >> > > > > > > >> > >> An interesting use case that I am holding onto is
> >> the
> >> > >> > > ability
> >> > >> > > > > for
> >> > >> > > > > > > >> > >> publishers to register via a single URL
> >> (registration
> >> > >> > > > > endpoint),
> >> > >> > > > > > > but
> >> > >> > > > > > > >> > >> be sent a URL back to post to a different process
> >> space
> >> > >> > for
> >> > >> > > > > actual
> >> > >> > > > > > > >> > >> publishing.  The same is true on the subscriber
> >> front.
> >> > >> > > > >  Currently,
> >> > >> > > > > > > the
> >> > >> > > > > > > >> > >> Camel/EIP infrastructure abstracts this because
> >> > >> different
> >> > >> > > > > > > components
> >> > >> > > > > > > >> > >> deployed in different process spaces handling the
> >> route
> >> > >> > > > > creation
> >> > >> > > > > > > can
> >> > >> > > > > > > >> > >> just be bolted onto a running Streams instance
> >> without
> >> > >> > new
> >> > >> > > > > > > subs/pubs
> >> > >> > > > > > > >> > >> behaving any differently than existing.  This
> >> seems to
> >> > >> > be a
> >> > >> > > > > > > >> > >> potentially critical scaling point.  Is there a
> >> way to
> >> > >> do
> >> > >> > > > this
> >> > >> > > > > with
> >> > >> > > > > > > >> > >> the Spring solution?
> >> > >> > > > > > > >> > >>
> >> > >> > > > > > > >> > >> The persistence point is a good one, though I
> >> would
> >> > >> > > classify
> >> > >> > > > > that
> >> > >> > > > > > > as
> >> > >> > > > > > > >> > >> "not implemented" vs "not possible" (not that you
> >> > >> were).
> >> > >> > > > > > > >> > >>
> >> > >> > > > > > > >> > >> I'm not married to Camel, I just like the EIP
> >> approach
> >> > >> to
> >> > >> > > > > building
> >> > >> > > > > > > >> > >> something that is ultimately a messaging system.
> >>  There
> >> > >> > are
> >> > >> > > > > known
> >> > >> > > > > > > >> > >> patterns that solve at least a subset of the
> >> problems
> >> > >> > > Streams
> >> > >> > > > > is
> >> > >> > > > > > > >> > >> trying to solve and implementations that can
> >> handle the
> >> > >> > > load
> >> > >> > > > > and
> >> > >> > > > > > > I'll
> >> > >> > > > > > > >> > >> reiterate flexibility == complexity almost always.
> >> > >> > > > > > > >> > >>
> >> > >> > > > > > > >> > >> It comes right back to the central question: Do
> >> you
> >> > >> want
> >> > >> > > > > > > flexibility
> >> > >> > > > > > > >> > >> or simplicity?  It doesn't have to be black and
> >> white
> >> > >> > > either
> >> > >> > > > I
> >> > >> > > > > > > don't
> >> > >> > > > > > > >> > >> think...
> >> > >> > > > > > > >> > >>
> >> > >> > > > > > > >> > >> More pointedly: Where should we give up
> >> flexibility for
> >> > >> > > > > > > simplicity?  I
> >> > >> > > > > > > >> > >> read that Danny is advocating for the simplicity
> >> at
> >> > >> > > > > registration to
> >> > >> > > > > > > >> > >> give up flexibility at registration, but
> >> retaining the
> >> > >> > > inner
> >> > >> > > > > > > "guts" of
> >> > >> > > > > > > >> > >> EIP/messaging?  Thoughts?
> >> > >> > > > > > > >> > >>
> >> > >> > > > > > > >> > >> On Wed, Oct 30, 2013 at 11:37 AM, Danny Sullivan <
> >> > >> > > > > > > dsullivan7@hotmail.com> wrote:
> >> > >> > > > > > > >> > >> > My argument is not for the IoC pattern as that
> >> can be
> >> > >> > > (and
> >> > >> > > > > has
> >> > >> > > > > > > been) implemented alongside Camel. My main argument is
> >> that the
> >> > >> > > > syntax
> >> > >> > > > > at
> >> > >> > > > > > > the entry point is not only familiar but much simpler.
> >> This
> >> > >> > > wouldn't
> >> > >> > > > > be a
> >> > >> > > > > > > very strong argument if the Camel implementation wasn't
> >> much
> >> > >> more
> >> > >> > > > > > > complicated but I feel that it is the case. Also, looking
> >> > >> toward
> >> > >> > > the
> >> > >> > > > > > > future, if the server is restarted, in-routes are lost in
> >> > >> Camel.
> >> > >> > > The
> >> > >> > > > > way to
> >> > >> > > > > > > curb this is to persist the dynamic routes that Camel
> >> creates,
> >> > >> > and
> >> > >> > > > > then on
> >> > >> > > > > > > start up pull every one of these routes and recreate a
> >> dynamic
> >> > >> > > route
> >> > >> > > > > for
> >> > >> > > > > > > each one. Not only is this much easier to implement using
> >> the
> >> > >> > > Spring
> >> > >> > > > > web
> >> > >> > > > > > > implementation, but it already has been implemented and
> >> you can
> >> > >> > try
> >> > >> > > > it
> >> > >> > > > > by
> >> > >> > > > > > > checking out the webservice branch, registering a
> >> subscriber,
> >> > >> > > > > restarting
> >> > >> > > > > > > tomcat, and using the same url you had before. This will
> >> allow
> >> > >> > > > > subscribers
> >> > >> > > > > > > to hang on to their urls once they register. (the same is
> >> true
> >> > >> > for
> >> > >> > > > > > > publishers: you can post via the same url after restarting
> >> > >> > tomcat)
> >> > >> > > > > > > >> > >> >
> >> > >> > > > > > > >> > >> >> Date: Wed, 30 Oct 2013 11:00:21 -0400
> >> > >> > > > > > > >> > >> >> Subject: Re: [DISCUSS] Switching Streams from
> >> Camel
> >> > >> > > > > deployment
> >> > >> > > > > > > to .war deployment
> >> > >> > > > > > > >> > >> >> From: jletourneau80@gmail.com
> >> > >> > > > > > > >> > >> >> To: dev@streams.incubator.apache.org
> >> > >> > > > > > > >> > >> >>
> >> > >> > > > > > > >> > >> >> To be fair, while the current implementation is
> >> > >> > heavily
> >> > >> > > > > > > camel-based,
> >> > >> > > > > > > >> > >> >> all of the interfaces related to Streams
> >> > >> functionality
> >> > >> > > are
> >> > >> > > > > not.
> >> > >> > > > > > >  The
> >> > >> > > > > > > >> > >> >> current model maps to what Matt has outlined
> >> in my
> >> > >> > > > opinion,
> >> > >> > > > > > > though
> >> > >> > > > > > > >> > >> >> packing names etc. probably don't follow that
> >> exact
> >> > >> > > > pattern.
> >> > >> > > > > > > >> > >> >>
> >> > >> > > > > > > >> > >> >> With regards to the complexity and different
> >> > >> > components
> >> > >> > > in
> >> > >> > > > > the
> >> > >> > > > > > > >> > >> >> registration process, this was a cut at the
> >> > >> > abstraction
> >> > >> > > > > based
> >> > >> > > > > > > on the
> >> > >> > > > > > > >> > >> >> assumption that different implementations may
> >> be
> >> > >> > plugged
> >> > >> > > > in
> >> > >> > > > > and
> >> > >> > > > > > > in
> >> > >> > > > > > > >> > >> >> fact may live on different processor space
> >> (ie. a
> >> > >> > > polling
> >> > >> > > > > > > publisher vs
> >> > >> > > > > > > >> > >> >> a push publisher may be instantiated on
> >> different
> >> > >> > > servers
> >> > >> > > > > but
> >> > >> > > > > > > the
> >> > >> > > > > > > >> > >> >> registration URL is staticly defined).
> >> > >> > > > > > > >> > >> >>
> >> > >> > > > > > > >> > >> >> Is the main argument I am seeing for Spring
> >>  the
> >> > >> > > > > familiarity of
> >> > >> > > > > > > its
> >> > >> > > > > > > >> > >> >> IoC pattern implementation and syntax at the
> >> entry
> >> > >> > > point?
> >> > >> > > > > > > >> > >> >>
> >> > >> > > > > > > >> > >> >> On Wed, Oct 30, 2013 at 10:53 AM, Danny
> >> Sullivan <
> >> > >> > > > > > > dsullivan7@hotmail.com> wrote:
> >> > >> > > > > > > >> > >> >> > Could you clarify whether the same entry
> >> points
> >> > >> > would
> >> > >> > > > > exist
> >> > >> > > > > > > for the camel implementation of the core (implementing the
> >> > >> > > "process"
> >> > >> > > > > > > method/ using a DynammicRouteBuilder) or would the
> >> webservice
> >> > >> be
> >> > >> > > the
> >> > >> > > > > sole
> >> > >> > > > > > > entry point to Streams and after it enters would it hand
> >> it off
> >> > >> > to
> >> > >> > > > > Camel?
> >> > >> > > > > > > And what would be the entry point for the Storm
> >> implementation?
> >> > >> > > > > > > >> > >> >> > -Danny
> >> > >> > > > > > > >> > >> >> >
> >> > >> > > > > > > >> > >> >> >> Date: Wed, 30 Oct 2013 10:13:04 -0400
> >> > >> > > > > > > >> > >> >> >> Subject: Re: [DISCUSS] Switching Streams
> >> from
> >> > >> Camel
> >> > >> > > > > > > deployment to .war deployment
> >> > >> > > > > > > >> > >> >> >> From: m.ben.franklin@gmail.com
> >> > >> > > > > > > >> > >> >> >> To: dev@streams.incubator.apache.org
> >> > >> > > > > > > >> > >> >> >>
> >> > >> > > > > > > >> > >> >> >> On Tue, Oct 29, 2013 at 2:55 PM, Danny
> >> Sullivan <
> >> > >> > > > > > > dsullivan7@hotmail.com>wrote:
> >> > >> > > > > > > >> > >> >> >>
> >> > >> > > > > > > >> > >> >> >> > My case for switching from OSGi is for
> >> > >> simplicity
> >> > >> > > in
> >> > >> > > > > > > design. To follow the
> >> > >> > > > > > > >> > >> >> >> > path of an activity through Streams in the
> >> > >> > > > webservice,
> >> > >> > > > > > > there is one main
> >> > >> > > > > > > >> > >> >> >> > things the developer needs to understand:
> >> > >> > > > > > > >> > >> >> >> > The @RequestMapping annotation specifies
> >> the
> >> > >> HTTP
> >> > >> > > > entry
> >> > >> > > > > > > point
> >> > >> > > > > > > >> > >> >> >> > There are 4 @RequestMapping annotations
> >> that
> >> > >> > > > > correspond to
> >> > >> > > > > > > each of the 4
> >> > >> > > > > > > >> > >> >> >> > ways a user enters the application:
> >> > >> registering a
> >> > >> > > > > > > publisher, registering a
> >> > >> > > > > > > >> > >> >> >> > subscriber, publishing activity, and
> >> getting an
> >> > >> > > > > activity
> >> > >> > > > > > > stream. Where
> >> > >> > > > > > > >> > >> >> >> > these are located in the source code can
> >> be
> >> > >> found
> >> > >> > > by
> >> > >> > > > > > > searching for the
> >> > >> > > > > > > >> > >> >> >> > paths specified in the documentation
> >> (search
> >> > >> for
> >> > >> > > > > > > "/publisherRegister",
> >> > >> > > > > > > >> > >> >> >> > "/subscriberRegister", "/publishActivity",
> >> > >> > > > > "/getActivity"
> >> > >> > > > > > > which will all
> >> > >> > > > > > > >> > >> >> >> > lead you to StreamsWebController.java).
> >> From
> >> > >> the
> >> > >> > > > > methods
> >> > >> > > > > > > that process
> >> > >> > > > > > > >> > >> >> >> > requests, the flow through the
> >> application is
> >> > >> > > through
> >> > >> > > > > > > methods which can be
> >> > >> > > > > > > >> > >> >> >> > understood by most Java programmers.
> >> > >> > > > > > > >> > >> >> >> > The flow of activities through the current
> >> > >> trunk
> >> > >> > > > > branch is
> >> > >> > > > > > > understood as
> >> > >> > > > > > > >> > >> >> >> > follows:
> >> > >> > > > > > > >> > >> >> >> > The string "/publisher/register" (the
> >> entry
> >> > >> point
> >> > >> > > to
> >> > >> > > > > > > register a publisher
> >> > >> > > > > > > >> > >> >> >> > specified in the documentation) is the
> >> value of
> >> > >> > the
> >> > >> > > > > > > >> > >> >> >> > consumer.registrationEndpoint property
> >> defined
> >> > >> in
> >> > >> > > > > > > streams.propertiesThe
> >> > >> > > > > > > >> > >> >> >> > camelContext.xml specifies an endpoint
> >> with the
> >> > >> > id
> >> > >> > > > > > > >> > >> >> >> > consumerRegistrationEndpoint to have a uri
> >> > >> equal
> >> > >> > to
> >> > >> > > > the
> >> > >> > > > > > > propertyThe
> >> > >> > > > > > > >> > >> >> >> > consumerRegistrationEndpoint routes from
> >> uri
> >> > >> > > > > > > direct:publisher register with
> >> > >> > > > > > > >> > >> >> >> > the bean activityRegistrationProcessor
> >> nested
> >> > >> in
> >> > >> > > > > between
> >> > >> > > > > > > the routeThe
> >> > >> > > > > > > >> > >> >> >> > streams-eip-applicationContext contains a
> >> bean
> >> > >> > with
> >> > >> > > > > the id
> >> > >> > > > > > > >> > >> >> >> > activityRegistrationProcessor created for
> >> the
> >> > >> > class
> >> > >> > > > > > > >> > >> >> >> > ActivityPublisherRegistrationProcessorThe
> >> > >> > exchange
> >> > >> > > > will
> >> > >> > > > > > > enter the "process"
> >> > >> > > > > > > >> > >> >> >> > method of the class
> >> > >> > > > > > > ActivityPublisherRegisitrationProcessor and that this
> >> > >> > > > > > > >> > >> >> >> > is because this class implements the
> >> > >> "Processor"
> >> > >> > > > > interface
> >> > >> > > > > > > provided by
> >> > >> > > > > > > >> > >> >> >> > CamelThe direct:add-publisher-route takes
> >> the
> >> > >> > > > exchange
> >> > >> > > > > > > output from the
> >> > >> > > > > > > >> > >> >> >> > "process" method and routes it to the
> >> > >> > > > > > > activityRegistrationProcessor
> >> > >> > > > > > > >> > >> >> >> > "register" method. The bean
> >> > >> > > > > activityRegistrationProcessor
> >> > >> > > > > > > is defined in the
> >> > >> > > > > > > >> > >> >> >> > streams-eip-osgi-component-import.xmlThe
> >> output
> >> > >> > > from
> >> > >> > > > > this
> >> > >> > > > > > > method is then
> >> > >> > > > > > > >> > >> >> >> > sent to the "createNewRouteForConsumer"
> >> method
> >> > >> of
> >> > >> > > > > > > activityConsumerRouter.
> >> > >> > > > > > > >> > >> >> >> > This method creates a new route for the
> >> newly
> >> > >> > > > > registered
> >> > >> > > > > > > publisher using
> >> > >> > > > > > > >> > >> >> >> > the private static final class
> >> > >> > > > > DynamicConsumerRouteBuilder
> >> > >> > > > > > > which is
> >> > >> > > > > > > >> > >> >> >> > required to extend RouteBuilder which is
> >> > >> provided
> >> > >> > > by
> >> > >> > > > > > > Camel. This
> >> > >> > > > > > > >> > >> >> >> > DynamicConsumerRouteBuilder contains
> >> several
> >> > >> > > methods:
> >> > >> > > > > > > >> > >> >> >> > getConsumerReceiveMethod() (which
> >> corresponds
> >> > >> to
> >> > >> > > > @Value
> >> > >> > > > > > > >> > >> >> >> > ${consumer.receiveMethod} which
> >> corresponds to
> >> > >> > > > > "receive"),
> >> > >> > > > > > > >> > >> >> >> > getConsumerSplitMethod() (@Value
> >> > >> > > > > > > ${consumer.splitMethod},"split"), and
> >> > >> > > > > > > >> > >> >> >> > getConsumerActivityQUri() (@Value
> >> > >> > > > > ${consumerActivityQUri},
> >> > >> > > > > > > >> > >> >> >> > "direct:activityQ"). This is different
> >> than the
> >> > >> > > > > > > camelContext.xml in that
> >> > >> > > > > > > >> > >> >> >> > the route is being created
> >> programmatically.
> >> > >> What
> >> > >> > > > this
> >> > >> > > > > is
> >> > >> > > > > > > doing is routing
> >> > >> > > > > > > >> > >> >> >> > input from the inroute url (which Camel
> >> does
> >> > >> > > > > automatically
> >> > >> > > > > > > through the
> >> > >> > > > > > > >> > >> >> >> > configure method which is required to be
> >> > >> > > overridden),
> >> > >> > > > > to
> >> > >> > > > > > > the "receive"
> >> > >> > > > > > > >> > >> >> >> > method of ActivityConsumer, then to the
> >> "split"
> >> > >> > > > method
> >> > >> > > > > of
> >> > >> > > > > > > ActivityConsumer,
> >> > >> > > > > > > >> > >> >> >> > and then to the "direct:activityQ" which
> >> if you
> >> > >> > > look
> >> > >> > > > > back
> >> > >> > > > > > > in the
> >> > >> > > > > > > >> > >> >> >> > camelContext.xml routes to the
> >> > >> > > > > "activemq:queue:activities"
> >> > >> > > > > > > which then
> >> > >> > > > > > > >> > >> >> >> > routes to "receiveExchange"
> >> > >> > > > > > > >> > >> >> >> > This is the process to register a
> >> publisher.
> >> > >> The
> >> > >> > > > > process
> >> > >> > > > > > > for registering a
> >> > >> > > > > > > >> > >> >> >> > subscriber is relatively the same though
> >> it
> >> > >> > > involves
> >> > >> > > > > > > separate classes with
> >> > >> > > > > > > >> > >> >> >> > their own private static final
> >> RouteBuilder
> >> > >> > class.
> >> > >> > > > > From my
> >> > >> > > > > > > perspective, the
> >> > >> > > > > > > >> > >> >> >> > two most difficult things with setting
> >> this
> >> > >> > project
> >> > >> > > > up
> >> > >> > > > > > > were understanding
> >> > >> > > > > > > >> > >> >> >> > that the "process" method of the class
> >> that
> >> > >> > > > implements
> >> > >> > > > > > > "Processor" is the
> >> > >> > > > > > > >> > >> >> >> > entry point and the
> >> DynamicConsumerRouteBuilder
> >> > >> > > > creates
> >> > >> > > > > > > the second entry
> >> > >> > > > > > > >> > >> >> >> > point (The 5th and last points). This
> >> made the
> >> > >> > > > project
> >> > >> > > > > > > very, VERY hard to
> >> > >> > > > > > > >> > >> >> >> > understand.
> >> > >> > > > > > > >> > >> >> >> > In addition to simplicity of design, the
> >> mvn
> >> > >> > clean
> >> > >> > > > > install
> >> > >> > > > > > > of the web
> >> > >> > > > > > > >> > >> >> >> > service project is much faster and small
> >> scale
> >> > >> > > > activity
> >> > >> > > > > > > publishing is also
> >> > >> > > > > > > >> > >> >> >> > faster (see my email about load testing).
> >> These
> >> > >> > are
> >> > >> > > > > minor
> >> > >> > > > > > > points though as
> >> > >> > > > > > > >> > >> >> >> > compilation has no effect on deployment.
> >> OSGi
> >> > >> > does
> >> > >> > > > add
> >> > >> > > > > the
> >> > >> > > > > > > benefit of
> >> > >> > > > > > > >> > >> >> >> > modularized programming which is valuable,
> >> > >> > though I
> >> > >> > > > > think
> >> > >> > > > > > > the added
> >> > >> > > > > > > >> > >> >> >> > complexity of Camel merits moving the
> >> project
> >> > >> > away
> >> > >> > > > from
> >> > >> > > > > > > this paradigm.
> >> > >> > > > > > > >> > >> >> >> >
> >> > >> > > > > > > >> > >> >> >>
> >> > >> > > > > > > >> > >> >> >> I agree that the project is pretty
> >> difficult to
> >> > >> > > > > understand
> >> > >> > > > > > > ATM.  I think
> >> > >> > > > > > > >> > >> >> >> what we need to do is think about what the
> >> > >> > > > > responsibilities
> >> > >> > > > > > > of the code are
> >> > >> > > > > > > >> > >> >> >> and allow for different implementations
> >> that are
> >> > >> > not
> >> > >> > > so
> >> > >> > > > > > > tightly coupled as
> >> > >> > > > > > > >> > >> >> >> they are now.  For instance, having worked
> >> with
> >> > >> > Storm
> >> > >> > > > to
> >> > >> > > > > > > ingest millions of
> >> > >> > > > > > > >> > >> >> >> activities a day, I personally would like
> >> to see
> >> > >> > > > streams
> >> > >> > > > > be
> >> > >> > > > > > > responsible for
> >> > >> > > > > > > >> > >> >> >> defining an over-arching orchestration
> >> model that
> >> > >> > can
> >> > >> > > > be
> >> > >> > > > > > > implemented within
> >> > >> > > > > > > >> > >> >> >> a single war or on top of a distributed
> >> system.
> >> > >> >  This
> >> > >> > > > > would
> >> > >> > > > > > > look something
> >> > >> > > > > > > >> > >> >> >> like the follows:
> >> > >> > > > > > > >> > >> >> >>
> >> > >> > > > > > > >> > >> >> >> |__ Streams-Core  (Base classes, interfaces,
> >> > >> > > utilities,
> >> > >> > > > > > > extensions, etc)
> >> > >> > > > > > > >> > >> >> >> |
> >> > >> > > > > > > >> > >> >> >> |__ Streams-Storm (Storm implementation of
> >> the
> >> > >> > core)
> >> > >> > > > > > > >> > >> >> >> |
> >> > >> > > > > > > >> > >> >> >> |__ Streams-Camel (Camel implementation of
> >> the
> >> > >> > core)
> >> > >> > > > > > > >> > >> >> >> |
> >> > >> > > > > > > >> > >> >> >> |__ Streams-WS (Web service implementation)
> >> > >> > > > > > > >> > >> >> >>
> >> > >> > > > > > > >> > >> >> >>
> >> > >> > > > > > > >> > >> >> >> > Danny
> >> > >> > > > > > > >> > >> >> >> >
> >> > >> > > > > > > >> > >> >> >> >
> >> > >> > > > > > > >> > >> >> >> > > From: dsullivan7@hotmail.com
> >> > >> > > > > > > >> > >> >> >> > > To: dev@streams.incubator.apache.org
> >> > >> > > > > > > >> > >> >> >> > > Subject: [DISCUSS] Switching Streams
> >> from
> >> > >> Camel
> >> > >> > > > > > > deployment to .war
> >> > >> > > > > > > >> > >> >> >> > deployment
> >> > >> > > > > > > >> > >> >> >> > > Date: Tue, 29 Oct 2013 11:07:39 -0400
> >> > >> > > > > > > >> > >> >> >> > >
> >> > >> > > > > > > >> > >> >> >> > > The discussion thread for switching
> >> Streams
> >> > >> > from
> >> > >> > > > > > > Camel/osgi/Servicemix
> >> > >> > > > > > > >> > >> >> >> > to a single .war deployment
> >> > >> > > > > > > >> > >> >> >> >
> >> > >> > > > > > > >> > >> >> >> >
> >> > >> > > > > > > >> > >> >> >
> >> > >> > > > > > > >> > >> >
> >> > >> > > > > > > >> > >
> >> > >> > > > > > > >>
> >> > >> > > > > > > >
> >> > >> > > > > > >
> >> > >> > > > >
> >> > >> > > > >
> >> > >> > > >
> >> > >> > >
> >> > >> >
> >> > >>
> >>
> >>
 		 	   		  
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message