streams-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason Letourneau <jletournea...@gmail.com>
Subject Re: [DISCUSS] Switching Streams from Camel deployment to .war deployment
Date Fri, 01 Nov 2013 16:53:28 GMT
Nice Chris - well put.  Streams isn't an application (or shouldn't be
thought of as one) it is a system that potentially is part of a system
of systems.

To punctuate your example, I also like to recall the classic EIP
banking/loan example [3]

[3] http://www.eaipatterns.com/ComposedMessagingExample.html

On Fri, Nov 1, 2013 at 12:48 PM, Chris Geer <chris@cxtsoftware.com> wrote:
> On Fri, Nov 1, 2013 at 6:56 AM, Danny Sullivan <dsullivan7@hotmail.com>wrote:
>
>> Perhaps it would be best for me to take a step back and ask what problem
>> are we trying to solve by using Camel? My understanding is that it is to
>> allow distribution of Streams to multiple servers. It would be helpful to
>> me to see another open source application that has this deployment
>> structure (it doesn't have to use Camel, just an application that has
>> multiple jars communicating with one another). Does anyone have any
>> suggestions of software for me to look at?
>>
>
> Danny, I wish I could just show you my system we are building. Honestly I
> think you are looking at this the wrong way. You are looking at it from a
> software development perspective (jars, protocols) and not from a
> business/system perspective (messages, components). So let me use an
> example.
>
> Let's say I have a business that takes order for widgets. I want the order
> taking to be streamlined for the user. To make that happen my order entry
> system does some basic checks when the user is placing the order and if
> those pass it saves the order to the database. To kick off the other
> processing, whenever I save an order to the database I sent a message to a
> Topic (pub/sub) saying either an order was created or an order was updated.
> Now at this point there is a lot more work I have to do to get the order
> shipped to the customer but it's an async action to the order taking. At
> this point the order has been placed and the customer can go about their
> business.
>
> I also have a component I wrote that listens to the Topic for "Order
> Created" messages. It takes each message, retrieves the order does some
> inventory processing to get the order ready to ship. At this point the
> order is complete and shipped to the customer.
>
> Now, in this simple example you could argue that the order entry system
> could just call the order processing component directly but that would
> preclude some extensibility in the future.
>
> A few months down the road I get a requirement to be able to notify the
> sales team whenever an order is placed by a certain set of customers. In a
> traditional model I'd have to modify the order entry component and put in
> logic to handle that. In this architecture all I have to do is build a
> different component that listens to that same Topic for order by certain
> customers. When it gets a notification it can then do the logic to send the
> message. The beauty of this is the order entry system doesn't' have to be
> modified and there is no risk of inserting a bug into that core process.
>
> So you ask, where does camel come in here. It could be used in lots of
> places but the natural fit is for the shipment processing component to be a
> camel route that listens to the Topic and calls a java object when messages
> come in. Also for the notification components, the whole thing could
> probably be a camel route (0 Java code).
>
> This is a pretty simple example and there are lots of other benefits I
> didn't talk about like redundancy and deployment flexibility.
>
> Does that help?
>
>>
>> > Date: Thu, 31 Oct 2013 20:23:26 -0400
>> > Subject: Re: [DISCUSS] Switching Streams from Camel deployment to .war
>> deployment
>> > From: jletourneau80@gmail.com
>> > To: dev@streams.incubator.apache.org
>> >
>> > So Danny in case it was lost in the ensuing transport discussion,
>> > Camel abstracts the transport, so you don't have to care if its http
>> > or whatever, only about the message format.  This is helpful if you
>> > care about plugging in components that DO care about the transport of
>> > course.  To answer your other questions, the downside to having the
>> > components all work via http (and not using messaging/Camel) is that
>> > everything immediately becomes point to point, which isn't a
>> > performance issue in its own right but limits scaling to more than one
>> > component being at the receiving end/fulfillment for instance as we've
>> > been discussing as being optimal.
>> >
>> > On Thu, Oct 31, 2013 at 5:38 PM, Matt Franklin <m.ben.franklin@gmail.com>
>> wrote:
>> > > On Thu, Oct 31, 2013 at 5:34 PM, Chris Geer <chris@cxtsoftware.com>
>> wrote:
>> > >
>> > >> On Thu, Oct 31, 2013 at 2:32 PM, Matt Franklin <
>> m.ben.franklin@gmail.com
>> > >> >wrote:
>> > >>
>> > >> > On Thu, Oct 31, 2013 at 5:28 PM, Chris Geer <chris@cxtsoftware.com>
>> > >> wrote:
>> > >> >
>> > >> > > On Thu, Oct 31, 2013 at 2:23 PM, Matt Franklin <
>> > >> m.ben.franklin@gmail.com
>> > >> > > >wrote:
>> > >> > >
>> > >> > > > On Thu, Oct 31, 2013 at 5:04 PM, Danny Sullivan <
>> > >> > dsullivan7@hotmail.com
>> > >> > > > >wrote:
>> > >> > > >
>> > >> > > > > I'm not quite following this, so I apologize. What I'm trying
>> to do
>> > >> > is
>> > >> > > > > programmatically make a request to a jar running on a
>> separate jvm
>> > >> > and
>> > >> > > > get
>> > >> > > > > the response from that call all within the same method.
>> Similar to
>> > >> > this
>> > >> > > > > http request:
>> > >> > > > >
>> > >> > > > > HttpGet httpget = new HttpGet();
>> > >> > > > >
>> > >> > > > > httpget.setURI(new URI("www.streams-persistence.com"));
>> > >> > > > >
>> > >> > > > > CloseableHttpResponse response = httpclient.execute(httpget);
>> > >> > > > >
>> > >> > > > > //do stuff with the response...
>> > >> > > > >
>> > >> > > > >
>> > >> > > > >
>> > >> > > > >
>> > >> > > > > I imagine that this would translate to something in Camel
>> similar
>> > >> to
>> > >> > > > this:
>> > >> > > > >
>> > >> > > > >
>> > >> > > > > <route>
>> > >> > > > >
>> > >> > > > > <from uri="bean:subscriberService?method=getAllSubscribers"/>
>> > >> > > > >
>> > >> > > > > <inOut
>> > >> > > > >
>> > >> > > >
>> > >> > >
>> > >> >
>> > >>
>> uri="addressToStreamsPersistenceJar?method=selectAllSubscribersFromDatabase"/>
>> > >> > > > >
>> > >> > > > > </route>
>> > >> > > > >
>> > >> > > >
>> > >> > > > Camel uses AMQP as a messaging system so that you don't have to
>> > >> create
>> > >> > > HTTP
>> > >> > > > requests between sources, though it is possible to do so.  In
>> Storm,
>> > >> we
>> > >> > > > usually try to use a buffer system like Kafka to do the same.
>> > >> > > >
>> > >> > >
>> > >> > > Camel can use AMQP through ActiveMQ.
>> > >> > >
>> > >> > >
>> > >> > Can being the operative word.  I haven't done much with Camel.  What
>> > >> other
>> > >> > transports are available.
>> > >> >
>> > >> > OpenWire (native ActiveMQ)
>> > >> AMQP
>> > >> STOMP
>> > >> HTTP (can be a server or client)
>> > >> Mail
>> > >> Amazon SQS
>> > >> XMPP
>> > >> ....hundred or so more
>> > >>
>> > >> http://camel.apache.org/components.html
>> > >
>> > >
>> > >
>> > > Nice.  I was expecting RTFM, so thanks for catering to my laziness.
>> > >
>> > >
>> > >>
>> > >>
>> > >> >
>> > >> > > >
>> > >> > > >
>> > >> > > > >
>> > >> > > > >
>> > >> > > > > But it is unclear what the actual implementation would be.
>> > >> > > > >
>> > >> > > > >
>> > >> > > > > This actually brings me to another suggestion. Would there be
>> a big
>> > >> > > > > performance impact to have communication between the software
>> > >> > > components
>> > >> > > > > occur between http? Say the 5 software components I outlined
>> > >> earlier
>> > >> > > were
>> > >> > > > > packaged as 5 separate wars. These wars could communicate
>> with each
>> > >> > > other
>> > >> > > > > via get a post requests. This sounds unconventional offhand
>> so I'd
>> > >> > like
>> > >> > > > to
>> > >> > > > > hear some thoughts on it.
>> > >> > > > >
>> > >> > > >
>> > >> > > > It is most certainly possible, but IMO probably not the best
>> option
>> > >> for
>> > >> > > > success.  Protocols like PubSubHubbub use HTTP for a message
>> > >> transport.
>> > >> > > >
>> > >> > > >
>> > >> > > > > -Danny
>> > >> > > > >
>> > >> > > > > > Date: Thu, 31 Oct 2013 13:28:41 -0700
>> > >> > > > > > Subject: Re: [DISCUSS] Switching Streams from Camel
>> deployment to
>> > >> > > .war
>> > >> > > > > deployment
>> > >> > > > > > From: chris@cxtsoftware.com
>> > >> > > > > > To: dev@streams.incubator.apache.org
>> > >> > > > > >
>> > >> > > > > > Or Content Enricher [2]
>> > >> > > > > >
>> > >> > > > > > [2] http://camel.apache.org/content-enricher.html
>> > >> > > > > >
>> > >> > > > > >
>> > >> > > > > > On Thu, Oct 31, 2013 at 12:37 PM, Jason Letourneau
>> > >> > > > > > <jletourneau80@gmail.com>wrote:
>> > >> > > > > >
>> > >> > > > > > > check out the link here[1]
>> > >> > > > > > >
>> > >> > > > > > >
>> > >> > > > > > > [1]http://camel.apache.org/request-reply.html
>> > >> > > > > > >
>> > >> > > > > > > On Thu, Oct 31, 2013 at 2:32 PM, Danny Sullivan <
>> > >> > > > > dsullivan7@hotmail.com>
>> > >> > > > > > > wrote:
>> > >> > > > > > > > I have a quick Camel question that I arrived at in the
>> > >> > > > > implementation of
>> > >> > > > > > > these new components:
>> > >> > > > > > > >
>> > >> > > > > > > > Lets say I have a method in streams-activity.jar that
>> needs
>> > >> all
>> > >> > > > > > > subscribers in the database. This would require a call to
>> the
>> > >> > > > > > > streams-persistence.jar. So far, I've seen camel used
>> mostly
>> > >> for
>> > >> > > > > passing
>> > >> > > > > > > data through the application, but not for making a single
>> > >> > > > > request-reponse
>> > >> > > > > > > from within a method. How can I use Camel to get a list
>> of all
>> > >> > > > > subscribers
>> > >> > > > > > > in the streams-persistence.jar from the
>> streams-activity.jar?
>> > >> > > > > > > >
>> > >> > > > > > > >> From: dsullivan7@hotmail.com
>> > >> > > > > > > >> To: dev@streams.incubator.apache.org
>> > >> > > > > > > >> Subject: RE: [DISCUSS] Switching Streams from Camel
>> > >> deployment
>> > >> > > to
>> > >> > > > > .war
>> > >> > > > > > > deployment
>> > >> > > > > > > >> Date: Thu, 31 Oct 2013 08:40:44 -0400
>> > >> > > > > > > >>
>> > >> > > > > > > >> Excellent, I'll write up something as a proof of
>> concept and
>> > >> > we
>> > >> > > > can
>> > >> > > > > > > discuss further to make sure everything is vanilla.
>> > >> > > > > > > >>
>> > >> > > > > > > >> > Date: Wed, 30 Oct 2013 17:01:28 -0400
>> > >> > > > > > > >> > Subject: Re: [DISCUSS] Switching Streams from Camel
>> > >> > deployment
>> > >> > > > to
>> > >> > > > > > > .war deployment
>> > >> > > > > > > >> > From: jletourneau80@gmail.com
>> > >> > > > > > > >> > To: dev@streams.incubator.apache.org
>> > >> > > > > > > >> >
>> > >> > > > > > > >> > That sounds pretty promising to me.
>> > >> > > > > > > >> >
>> > >> > > > > > > >> > On Wed, Oct 30, 2013 at 2:38 PM, Danny Sullivan <
>> > >> > > > > > > dsullivan7@hotmail.com> wrote:
>> > >> > > > > > > >> > > Thanks for the feedback. You have an interesting
>> point
>> > >> > about
>> > >> > > > the
>> > >> > > > > > > url linking to a separate processing space. Let me tie my
>> > >> answer
>> > >> > > into
>> > >> > > > > your
>> > >> > > > > > > last question about "advocating for the simplicity at
>> > >> > registration
>> > >> > > to
>> > >> > > > > give
>> > >> > > > > > > up flexibility at registration, but retaining the inner
>> "guts"
>> > >> of
>> > >> > > > > > > EIP/messaging". Consider a new architecture:
>> > >> > > > > > > >> > >
>> > >> > > > > > > >> > > streams-web.war: single entry point to
>> application, but
>> > >> > > > > functions
>> > >> > > > > > > ONLY as an entry point. From here Camel routes the
>> incoming
>> > >> > > requests
>> > >> > > > > to 4
>> > >> > > > > > > separate jarssubscriber-registration.jar: subscriber
>> > >> registration
>> > >> > > > > > > publisher-registration.jar: publisher
>> registrationactivity.jar:
>> > >> > > > returns
>> > >> > > > > > > activity (also contains subscriber warehouse and storm
>> activity
>> > >> > > > > > > aggregator)publish.jar: publishes
>> > >> activitystreams-cassandra.jar:
>> > >> > > the
>> > >> > > > > above
>> > >> > > > > > > 4 jars would all have a hook into this jar which would
>> function
>> > >> > as
>> > >> > > a
>> > >> > > > > hook
>> > >> > > > > > > onto the database. Each jar would have camel route output
>> to
>> > >> this
>> > >> > > > jar.
>> > >> > > > > > > >> > >
>> > >> > > > > > > >> > > In this implementation, Camel would no longer be
>> the
>> > >> entry
>> > >> > > and
>> > >> > > > > exit
>> > >> > > > > > > point of a client to the application, but would handle the
>> > >> > > > > communication
>> > >> > > > > > > between components. The flow of activity through the
>> > >> application
>> > >> > > > would
>> > >> > > > > be
>> > >> > > > > > > method based in each jar. This would allow deployment on
>> up to
>> > >> 6
>> > >> > > > > different
>> > >> > > > > > > process spaces. However, this does not address that there
>> is a
>> > >> > > single
>> > >> > > > > > > server entry point, but I'm not sure if it was a concern
>> in the
>> > >> > > first
>> > >> > > > > place.
>> > >> > > > > > > >> > >
>> > >> > > > > > > >> > > My argument, at its basis, is that we should move
>> away
>> > >> > from
>> > >> > > > > using
>> > >> > > > > > > Camel as the entry point to the application. I would be
>> happy
>> > >> to
>> > >> > > > > maintain
>> > >> > > > > > > messaging between components.
>> > >> > > > > > > >> > >
>> > >> > > > > > > >> > >> Date: Wed, 30 Oct 2013 12:32:55 -0400
>> > >> > > > > > > >> > >> Subject: Re: [DISCUSS] Switching Streams from
>> Camel
>> > >> > > > deployment
>> > >> > > > > to
>> > >> > > > > > > .war deployment
>> > >> > > > > > > >> > >> From: jletourneau80@gmail.com
>> > >> > > > > > > >> > >> To: dev@streams.incubator.apache.org
>> > >> > > > > > > >> > >>
>> > >> > > > > > > >> > >> An interesting use case that I am holding onto is
>> the
>> > >> > > ability
>> > >> > > > > for
>> > >> > > > > > > >> > >> publishers to register via a single URL
>> (registration
>> > >> > > > > endpoint),
>> > >> > > > > > > but
>> > >> > > > > > > >> > >> be sent a URL back to post to a different process
>> space
>> > >> > for
>> > >> > > > > actual
>> > >> > > > > > > >> > >> publishing.  The same is true on the subscriber
>> front.
>> > >> > > > >  Currently,
>> > >> > > > > > > the
>> > >> > > > > > > >> > >> Camel/EIP infrastructure abstracts this because
>> > >> different
>> > >> > > > > > > components
>> > >> > > > > > > >> > >> deployed in different process spaces handling the
>> route
>> > >> > > > > creation
>> > >> > > > > > > can
>> > >> > > > > > > >> > >> just be bolted onto a running Streams instance
>> without
>> > >> > new
>> > >> > > > > > > subs/pubs
>> > >> > > > > > > >> > >> behaving any differently than existing.  This
>> seems to
>> > >> > be a
>> > >> > > > > > > >> > >> potentially critical scaling point.  Is there a
>> way to
>> > >> do
>> > >> > > > this
>> > >> > > > > with
>> > >> > > > > > > >> > >> the Spring solution?
>> > >> > > > > > > >> > >>
>> > >> > > > > > > >> > >> The persistence point is a good one, though I
>> would
>> > >> > > classify
>> > >> > > > > that
>> > >> > > > > > > as
>> > >> > > > > > > >> > >> "not implemented" vs "not possible" (not that you
>> > >> were).
>> > >> > > > > > > >> > >>
>> > >> > > > > > > >> > >> I'm not married to Camel, I just like the EIP
>> approach
>> > >> to
>> > >> > > > > building
>> > >> > > > > > > >> > >> something that is ultimately a messaging system.
>>  There
>> > >> > are
>> > >> > > > > known
>> > >> > > > > > > >> > >> patterns that solve at least a subset of the
>> problems
>> > >> > > Streams
>> > >> > > > > is
>> > >> > > > > > > >> > >> trying to solve and implementations that can
>> handle the
>> > >> > > load
>> > >> > > > > and
>> > >> > > > > > > I'll
>> > >> > > > > > > >> > >> reiterate flexibility == complexity almost always.
>> > >> > > > > > > >> > >>
>> > >> > > > > > > >> > >> It comes right back to the central question: Do
>> you
>> > >> want
>> > >> > > > > > > flexibility
>> > >> > > > > > > >> > >> or simplicity?  It doesn't have to be black and
>> white
>> > >> > > either
>> > >> > > > I
>> > >> > > > > > > don't
>> > >> > > > > > > >> > >> think...
>> > >> > > > > > > >> > >>
>> > >> > > > > > > >> > >> More pointedly: Where should we give up
>> flexibility for
>> > >> > > > > > > simplicity?  I
>> > >> > > > > > > >> > >> read that Danny is advocating for the simplicity
>> at
>> > >> > > > > registration to
>> > >> > > > > > > >> > >> give up flexibility at registration, but
>> retaining the
>> > >> > > inner
>> > >> > > > > > > "guts" of
>> > >> > > > > > > >> > >> EIP/messaging?  Thoughts?
>> > >> > > > > > > >> > >>
>> > >> > > > > > > >> > >> On Wed, Oct 30, 2013 at 11:37 AM, Danny Sullivan <
>> > >> > > > > > > dsullivan7@hotmail.com> wrote:
>> > >> > > > > > > >> > >> > My argument is not for the IoC pattern as that
>> can be
>> > >> > > (and
>> > >> > > > > has
>> > >> > > > > > > been) implemented alongside Camel. My main argument is
>> that the
>> > >> > > > syntax
>> > >> > > > > at
>> > >> > > > > > > the entry point is not only familiar but much simpler.
>> This
>> > >> > > wouldn't
>> > >> > > > > be a
>> > >> > > > > > > very strong argument if the Camel implementation wasn't
>> much
>> > >> more
>> > >> > > > > > > complicated but I feel that it is the case. Also, looking
>> > >> toward
>> > >> > > the
>> > >> > > > > > > future, if the server is restarted, in-routes are lost in
>> > >> Camel.
>> > >> > > The
>> > >> > > > > way to
>> > >> > > > > > > curb this is to persist the dynamic routes that Camel
>> creates,
>> > >> > and
>> > >> > > > > then on
>> > >> > > > > > > start up pull every one of these routes and recreate a
>> dynamic
>> > >> > > route
>> > >> > > > > for
>> > >> > > > > > > each one. Not only is this much easier to implement using
>> the
>> > >> > > Spring
>> > >> > > > > web
>> > >> > > > > > > implementation, but it already has been implemented and
>> you can
>> > >> > try
>> > >> > > > it
>> > >> > > > > by
>> > >> > > > > > > checking out the webservice branch, registering a
>> subscriber,
>> > >> > > > > restarting
>> > >> > > > > > > tomcat, and using the same url you had before. This will
>> allow
>> > >> > > > > subscribers
>> > >> > > > > > > to hang on to their urls once they register. (the same is
>> true
>> > >> > for
>> > >> > > > > > > publishers: you can post via the same url after restarting
>> > >> > tomcat)
>> > >> > > > > > > >> > >> >
>> > >> > > > > > > >> > >> >> Date: Wed, 30 Oct 2013 11:00:21 -0400
>> > >> > > > > > > >> > >> >> Subject: Re: [DISCUSS] Switching Streams from
>> Camel
>> > >> > > > > deployment
>> > >> > > > > > > to .war deployment
>> > >> > > > > > > >> > >> >> From: jletourneau80@gmail.com
>> > >> > > > > > > >> > >> >> To: dev@streams.incubator.apache.org
>> > >> > > > > > > >> > >> >>
>> > >> > > > > > > >> > >> >> To be fair, while the current implementation is
>> > >> > heavily
>> > >> > > > > > > camel-based,
>> > >> > > > > > > >> > >> >> all of the interfaces related to Streams
>> > >> functionality
>> > >> > > are
>> > >> > > > > not.
>> > >> > > > > > >  The
>> > >> > > > > > > >> > >> >> current model maps to what Matt has outlined
>> in my
>> > >> > > > opinion,
>> > >> > > > > > > though
>> > >> > > > > > > >> > >> >> packing names etc. probably don't follow that
>> exact
>> > >> > > > pattern.
>> > >> > > > > > > >> > >> >>
>> > >> > > > > > > >> > >> >> With regards to the complexity and different
>> > >> > components
>> > >> > > in
>> > >> > > > > the
>> > >> > > > > > > >> > >> >> registration process, this was a cut at the
>> > >> > abstraction
>> > >> > > > > based
>> > >> > > > > > > on the
>> > >> > > > > > > >> > >> >> assumption that different implementations may
>> be
>> > >> > plugged
>> > >> > > > in
>> > >> > > > > and
>> > >> > > > > > > in
>> > >> > > > > > > >> > >> >> fact may live on different processor space
>> (ie. a
>> > >> > > polling
>> > >> > > > > > > publisher vs
>> > >> > > > > > > >> > >> >> a push publisher may be instantiated on
>> different
>> > >> > > servers
>> > >> > > > > but
>> > >> > > > > > > the
>> > >> > > > > > > >> > >> >> registration URL is staticly defined).
>> > >> > > > > > > >> > >> >>
>> > >> > > > > > > >> > >> >> Is the main argument I am seeing for Spring
>>  the
>> > >> > > > > familiarity of
>> > >> > > > > > > its
>> > >> > > > > > > >> > >> >> IoC pattern implementation and syntax at the
>> entry
>> > >> > > point?
>> > >> > > > > > > >> > >> >>
>> > >> > > > > > > >> > >> >> On Wed, Oct 30, 2013 at 10:53 AM, Danny
>> Sullivan <
>> > >> > > > > > > dsullivan7@hotmail.com> wrote:
>> > >> > > > > > > >> > >> >> > Could you clarify whether the same entry
>> points
>> > >> > would
>> > >> > > > > exist
>> > >> > > > > > > for the camel implementation of the core (implementing the
>> > >> > > "process"
>> > >> > > > > > > method/ using a DynammicRouteBuilder) or would the
>> webservice
>> > >> be
>> > >> > > the
>> > >> > > > > sole
>> > >> > > > > > > entry point to Streams and after it enters would it hand
>> it off
>> > >> > to
>> > >> > > > > Camel?
>> > >> > > > > > > And what would be the entry point for the Storm
>> implementation?
>> > >> > > > > > > >> > >> >> > -Danny
>> > >> > > > > > > >> > >> >> >
>> > >> > > > > > > >> > >> >> >> Date: Wed, 30 Oct 2013 10:13:04 -0400
>> > >> > > > > > > >> > >> >> >> Subject: Re: [DISCUSS] Switching Streams
>> from
>> > >> Camel
>> > >> > > > > > > deployment to .war deployment
>> > >> > > > > > > >> > >> >> >> From: m.ben.franklin@gmail.com
>> > >> > > > > > > >> > >> >> >> To: dev@streams.incubator.apache.org
>> > >> > > > > > > >> > >> >> >>
>> > >> > > > > > > >> > >> >> >> On Tue, Oct 29, 2013 at 2:55 PM, Danny
>> Sullivan <
>> > >> > > > > > > dsullivan7@hotmail.com>wrote:
>> > >> > > > > > > >> > >> >> >>
>> > >> > > > > > > >> > >> >> >> > My case for switching from OSGi is for
>> > >> simplicity
>> > >> > > in
>> > >> > > > > > > design. To follow the
>> > >> > > > > > > >> > >> >> >> > path of an activity through Streams in the
>> > >> > > > webservice,
>> > >> > > > > > > there is one main
>> > >> > > > > > > >> > >> >> >> > things the developer needs to understand:
>> > >> > > > > > > >> > >> >> >> > The @RequestMapping annotation specifies
>> the
>> > >> HTTP
>> > >> > > > entry
>> > >> > > > > > > point
>> > >> > > > > > > >> > >> >> >> > There are 4 @RequestMapping annotations
>> that
>> > >> > > > > correspond to
>> > >> > > > > > > each of the 4
>> > >> > > > > > > >> > >> >> >> > ways a user enters the application:
>> > >> registering a
>> > >> > > > > > > publisher, registering a
>> > >> > > > > > > >> > >> >> >> > subscriber, publishing activity, and
>> getting an
>> > >> > > > > activity
>> > >> > > > > > > stream. Where
>> > >> > > > > > > >> > >> >> >> > these are located in the source code can
>> be
>> > >> found
>> > >> > > by
>> > >> > > > > > > searching for the
>> > >> > > > > > > >> > >> >> >> > paths specified in the documentation
>> (search
>> > >> for
>> > >> > > > > > > "/publisherRegister",
>> > >> > > > > > > >> > >> >> >> > "/subscriberRegister", "/publishActivity",
>> > >> > > > > "/getActivity"
>> > >> > > > > > > which will all
>> > >> > > > > > > >> > >> >> >> > lead you to StreamsWebController.java).
>> From
>> > >> the
>> > >> > > > > methods
>> > >> > > > > > > that process
>> > >> > > > > > > >> > >> >> >> > requests, the flow through the
>> application is
>> > >> > > through
>> > >> > > > > > > methods which can be
>> > >> > > > > > > >> > >> >> >> > understood by most Java programmers.
>> > >> > > > > > > >> > >> >> >> > The flow of activities through the current
>> > >> trunk
>> > >> > > > > branch is
>> > >> > > > > > > understood as
>> > >> > > > > > > >> > >> >> >> > follows:
>> > >> > > > > > > >> > >> >> >> > The string "/publisher/register" (the
>> entry
>> > >> point
>> > >> > > to
>> > >> > > > > > > register a publisher
>> > >> > > > > > > >> > >> >> >> > specified in the documentation) is the
>> value of
>> > >> > the
>> > >> > > > > > > >> > >> >> >> > consumer.registrationEndpoint property
>> defined
>> > >> in
>> > >> > > > > > > streams.propertiesThe
>> > >> > > > > > > >> > >> >> >> > camelContext.xml specifies an endpoint
>> with the
>> > >> > id
>> > >> > > > > > > >> > >> >> >> > consumerRegistrationEndpoint to have a uri
>> > >> equal
>> > >> > to
>> > >> > > > the
>> > >> > > > > > > propertyThe
>> > >> > > > > > > >> > >> >> >> > consumerRegistrationEndpoint routes from
>> uri
>> > >> > > > > > > direct:publisher register with
>> > >> > > > > > > >> > >> >> >> > the bean activityRegistrationProcessor
>> nested
>> > >> in
>> > >> > > > > between
>> > >> > > > > > > the routeThe
>> > >> > > > > > > >> > >> >> >> > streams-eip-applicationContext contains a
>> bean
>> > >> > with
>> > >> > > > > the id
>> > >> > > > > > > >> > >> >> >> > activityRegistrationProcessor created for
>> the
>> > >> > class
>> > >> > > > > > > >> > >> >> >> > ActivityPublisherRegistrationProcessorThe
>> > >> > exchange
>> > >> > > > will
>> > >> > > > > > > enter the "process"
>> > >> > > > > > > >> > >> >> >> > method of the class
>> > >> > > > > > > ActivityPublisherRegisitrationProcessor and that this
>> > >> > > > > > > >> > >> >> >> > is because this class implements the
>> > >> "Processor"
>> > >> > > > > interface
>> > >> > > > > > > provided by
>> > >> > > > > > > >> > >> >> >> > CamelThe direct:add-publisher-route takes
>> the
>> > >> > > > exchange
>> > >> > > > > > > output from the
>> > >> > > > > > > >> > >> >> >> > "process" method and routes it to the
>> > >> > > > > > > activityRegistrationProcessor
>> > >> > > > > > > >> > >> >> >> > "register" method. The bean
>> > >> > > > > activityRegistrationProcessor
>> > >> > > > > > > is defined in the
>> > >> > > > > > > >> > >> >> >> > streams-eip-osgi-component-import.xmlThe
>> output
>> > >> > > from
>> > >> > > > > this
>> > >> > > > > > > method is then
>> > >> > > > > > > >> > >> >> >> > sent to the "createNewRouteForConsumer"
>> method
>> > >> of
>> > >> > > > > > > activityConsumerRouter.
>> > >> > > > > > > >> > >> >> >> > This method creates a new route for the
>> newly
>> > >> > > > > registered
>> > >> > > > > > > publisher using
>> > >> > > > > > > >> > >> >> >> > the private static final class
>> > >> > > > > DynamicConsumerRouteBuilder
>> > >> > > > > > > which is
>> > >> > > > > > > >> > >> >> >> > required to extend RouteBuilder which is
>> > >> provided
>> > >> > > by
>> > >> > > > > > > Camel. This
>> > >> > > > > > > >> > >> >> >> > DynamicConsumerRouteBuilder contains
>> several
>> > >> > > methods:
>> > >> > > > > > > >> > >> >> >> > getConsumerReceiveMethod() (which
>> corresponds
>> > >> to
>> > >> > > > @Value
>> > >> > > > > > > >> > >> >> >> > ${consumer.receiveMethod} which
>> corresponds to
>> > >> > > > > "receive"),
>> > >> > > > > > > >> > >> >> >> > getConsumerSplitMethod() (@Value
>> > >> > > > > > > ${consumer.splitMethod},"split"), and
>> > >> > > > > > > >> > >> >> >> > getConsumerActivityQUri() (@Value
>> > >> > > > > ${consumerActivityQUri},
>> > >> > > > > > > >> > >> >> >> > "direct:activityQ"). This is different
>> than the
>> > >> > > > > > > camelContext.xml in that
>> > >> > > > > > > >> > >> >> >> > the route is being created
>> programmatically.
>> > >> What
>> > >> > > > this
>> > >> > > > > is
>> > >> > > > > > > doing is routing
>> > >> > > > > > > >> > >> >> >> > input from the inroute url (which Camel
>> does
>> > >> > > > > automatically
>> > >> > > > > > > through the
>> > >> > > > > > > >> > >> >> >> > configure method which is required to be
>> > >> > > overridden),
>> > >> > > > > to
>> > >> > > > > > > the "receive"
>> > >> > > > > > > >> > >> >> >> > method of ActivityConsumer, then to the
>> "split"
>> > >> > > > method
>> > >> > > > > of
>> > >> > > > > > > ActivityConsumer,
>> > >> > > > > > > >> > >> >> >> > and then to the "direct:activityQ" which
>> if you
>> > >> > > look
>> > >> > > > > back
>> > >> > > > > > > in the
>> > >> > > > > > > >> > >> >> >> > camelContext.xml routes to the
>> > >> > > > > "activemq:queue:activities"
>> > >> > > > > > > which then
>> > >> > > > > > > >> > >> >> >> > routes to "receiveExchange"
>> > >> > > > > > > >> > >> >> >> > This is the process to register a
>> publisher.
>> > >> The
>> > >> > > > > process
>> > >> > > > > > > for registering a
>> > >> > > > > > > >> > >> >> >> > subscriber is relatively the same though
>> it
>> > >> > > involves
>> > >> > > > > > > separate classes with
>> > >> > > > > > > >> > >> >> >> > their own private static final
>> RouteBuilder
>> > >> > class.
>> > >> > > > > From my
>> > >> > > > > > > perspective, the
>> > >> > > > > > > >> > >> >> >> > two most difficult things with setting
>> this
>> > >> > project
>> > >> > > > up
>> > >> > > > > > > were understanding
>> > >> > > > > > > >> > >> >> >> > that the "process" method of the class
>> that
>> > >> > > > implements
>> > >> > > > > > > "Processor" is the
>> > >> > > > > > > >> > >> >> >> > entry point and the
>> DynamicConsumerRouteBuilder
>> > >> > > > creates
>> > >> > > > > > > the second entry
>> > >> > > > > > > >> > >> >> >> > point (The 5th and last points). This
>> made the
>> > >> > > > project
>> > >> > > > > > > very, VERY hard to
>> > >> > > > > > > >> > >> >> >> > understand.
>> > >> > > > > > > >> > >> >> >> > In addition to simplicity of design, the
>> mvn
>> > >> > clean
>> > >> > > > > install
>> > >> > > > > > > of the web
>> > >> > > > > > > >> > >> >> >> > service project is much faster and small
>> scale
>> > >> > > > activity
>> > >> > > > > > > publishing is also
>> > >> > > > > > > >> > >> >> >> > faster (see my email about load testing).
>> These
>> > >> > are
>> > >> > > > > minor
>> > >> > > > > > > points though as
>> > >> > > > > > > >> > >> >> >> > compilation has no effect on deployment.
>> OSGi
>> > >> > does
>> > >> > > > add
>> > >> > > > > the
>> > >> > > > > > > benefit of
>> > >> > > > > > > >> > >> >> >> > modularized programming which is valuable,
>> > >> > though I
>> > >> > > > > think
>> > >> > > > > > > the added
>> > >> > > > > > > >> > >> >> >> > complexity of Camel merits moving the
>> project
>> > >> > away
>> > >> > > > from
>> > >> > > > > > > this paradigm.
>> > >> > > > > > > >> > >> >> >> >
>> > >> > > > > > > >> > >> >> >>
>> > >> > > > > > > >> > >> >> >> I agree that the project is pretty
>> difficult to
>> > >> > > > > understand
>> > >> > > > > > > ATM.  I think
>> > >> > > > > > > >> > >> >> >> what we need to do is think about what the
>> > >> > > > > responsibilities
>> > >> > > > > > > of the code are
>> > >> > > > > > > >> > >> >> >> and allow for different implementations
>> that are
>> > >> > not
>> > >> > > so
>> > >> > > > > > > tightly coupled as
>> > >> > > > > > > >> > >> >> >> they are now.  For instance, having worked
>> with
>> > >> > Storm
>> > >> > > > to
>> > >> > > > > > > ingest millions of
>> > >> > > > > > > >> > >> >> >> activities a day, I personally would like
>> to see
>> > >> > > > streams
>> > >> > > > > be
>> > >> > > > > > > responsible for
>> > >> > > > > > > >> > >> >> >> defining an over-arching orchestration
>> model that
>> > >> > can
>> > >> > > > be
>> > >> > > > > > > implemented within
>> > >> > > > > > > >> > >> >> >> a single war or on top of a distributed
>> system.
>> > >> >  This
>> > >> > > > > would
>> > >> > > > > > > look something
>> > >> > > > > > > >> > >> >> >> like the follows:
>> > >> > > > > > > >> > >> >> >>
>> > >> > > > > > > >> > >> >> >> |__ Streams-Core  (Base classes, interfaces,
>> > >> > > utilities,
>> > >> > > > > > > extensions, etc)
>> > >> > > > > > > >> > >> >> >> |
>> > >> > > > > > > >> > >> >> >> |__ Streams-Storm (Storm implementation of
>> the
>> > >> > core)
>> > >> > > > > > > >> > >> >> >> |
>> > >> > > > > > > >> > >> >> >> |__ Streams-Camel (Camel implementation of
>> the
>> > >> > core)
>> > >> > > > > > > >> > >> >> >> |
>> > >> > > > > > > >> > >> >> >> |__ Streams-WS (Web service implementation)
>> > >> > > > > > > >> > >> >> >>
>> > >> > > > > > > >> > >> >> >>
>> > >> > > > > > > >> > >> >> >> > Danny
>> > >> > > > > > > >> > >> >> >> >
>> > >> > > > > > > >> > >> >> >> >
>> > >> > > > > > > >> > >> >> >> > > From: dsullivan7@hotmail.com
>> > >> > > > > > > >> > >> >> >> > > To: dev@streams.incubator.apache.org
>> > >> > > > > > > >> > >> >> >> > > Subject: [DISCUSS] Switching Streams
>> from
>> > >> Camel
>> > >> > > > > > > deployment to .war
>> > >> > > > > > > >> > >> >> >> > deployment
>> > >> > > > > > > >> > >> >> >> > > Date: Tue, 29 Oct 2013 11:07:39 -0400
>> > >> > > > > > > >> > >> >> >> > >
>> > >> > > > > > > >> > >> >> >> > > The discussion thread for switching
>> Streams
>> > >> > from
>> > >> > > > > > > Camel/osgi/Servicemix
>> > >> > > > > > > >> > >> >> >> > to a single .war deployment
>> > >> > > > > > > >> > >> >> >> >
>> > >> > > > > > > >> > >> >> >> >
>> > >> > > > > > > >> > >> >> >
>> > >> > > > > > > >> > >> >
>> > >> > > > > > > >> > >
>> > >> > > > > > > >>
>> > >> > > > > > > >
>> > >> > > > > > >
>> > >> > > > >
>> > >> > > > >
>> > >> > > >
>> > >> > >
>> > >> >
>> > >>
>>
>>

Mime
View raw message