Return-Path: X-Original-To: apmail-streams-dev-archive@minotaur.apache.org Delivered-To: apmail-streams-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 52DB4101D0 for ; Sat, 2 Nov 2013 06:30:22 +0000 (UTC) Received: (qmail 10168 invoked by uid 500); 2 Nov 2013 06:30:21 -0000 Delivered-To: apmail-streams-dev-archive@streams.apache.org Received: (qmail 10122 invoked by uid 500); 2 Nov 2013 06:30:18 -0000 Mailing-List: contact dev-help@streams.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@streams.incubator.apache.org Delivered-To: mailing list dev@streams.incubator.apache.org Received: (qmail 10104 invoked by uid 99); 2 Nov 2013 06:30:14 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 02 Nov 2013 06:30:14 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy includes SPF record at spf.trusted-forwarder.org) Received: from [209.85.160.48] (HELO mail-pb0-f48.google.com) (209.85.160.48) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 02 Nov 2013 06:30:09 +0000 Received: by mail-pb0-f48.google.com with SMTP id mc17so5091775pbc.7 for ; Fri, 01 Nov 2013 23:29:49 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=hB8bFuLHyUmDlEkmGcDxG+uiV+9Hr+x/RjhTJYvRIZc=; b=RJNdrW8L1A3c5IfIzry1H/p7KtG/BWazvd5WmnLYpcRHBO+znjD5Fx+nV2gyFh+L8P 8mVf3mv8ApKnrhApGooBaNo4vzaxOT2kHSMJUQXtUmy0pXaOMYagHIwZIQD9Jt9RLy9u ycVtm18cGZuRKleyVZUo/VJ8Ea1GEUmgcmWhvxo4hnTyye7mpHUxHtXu9FoZ57ja+xy7 /1kstvnjt65TqtEaBOAz4jGG3RPLUxBW7yZ0nzpofSTMlAxq8AeLpHrI3P/nuevdp0zL oOyKN3+Rt2HW4X+9yAJQhCTJKcQliFR+DILBJaPskKlI8jwa5sF4uTy2APie7cHOycG3 +izA== X-Gm-Message-State: ALoCoQkKqQvZjqgj7sJWMOeVrMA3UZImrS1TDY8GIrlgkCWGGffm7RwdeARJRBy8J5fTtXdOydDu MIME-Version: 1.0 X-Received: by 10.68.125.198 with SMTP id ms6mr6760274pbb.98.1383373788852; Fri, 01 Nov 2013 23:29:48 -0700 (PDT) Received: by 10.70.132.104 with HTTP; Fri, 1 Nov 2013 23:29:48 -0700 (PDT) In-Reply-To: References: Date: Fri, 1 Nov 2013 23:29:48 -0700 Message-ID: Subject: Re: [DISCUSS] Switching Streams from Camel deployment to .war deployment From: Chris Geer To: "dev@streams.incubator.apache.org" Content-Type: multipart/alternative; boundary=047d7b2e4af246dfcd04ea2bcf3b X-Virus-Checked: Checked by ClamAV on apache.org --047d7b2e4af246dfcd04ea2bcf3b Content-Type: text/plain; charset=ISO-8859-1 On Fri, Nov 1, 2013 at 11:06 AM, Danny Sullivan wrote: > Excellent, thanks very much for the info. This does help me understand why > you would want to use EIP for development of certain systems. Because I > understand the value of this design I think it would be a good idea to keep > the current EIP implementation as the trunk branch and retract my request > to merge the webservice branch with trunk at this time. However, I think it > would also still be valuable to draw out the webservice branch and explore > Streams as an "application" rather than a "system", at least until there is > a crystal clear vision of what the direction should be. > > I think both of these application/systems will move in the same direction > and ultimately fulfill the initial design goal of a "lightweight (yet > scalable) server for ActivityStreams". To keep this goal in mind, I'd like > to keep the discussion going by posing a scenario. Consider that Streams is > completely developed. What are the answers to these questions in this > scenario: > > "What does Streams do?" > My 2 cents - Streams is a collection of components that can be used to add activity processing capabilities to your system/application. > > "How do I enable my application to support Streams?" > > I'm not sure it's really about enabling your application to support Streams, it's about using Streams to enable your system to support processing activities. Streams should give you the components which you can configure together to give you the right processing steps to get activity data in the format you need to power your system. > > Thanks very much for all the feedback, this has been a very good > discussion! > Danny > > > Date: Fri, 1 Nov 2013 12:53:28 -0400 > > Subject: Re: [DISCUSS] Switching Streams from Camel deployment to .war > deployment > > From: jletourneau80@gmail.com > > To: dev@streams.incubator.apache.org > > > > Nice Chris - well put. Streams isn't an application (or shouldn't be > > thought of as one) it is a system that potentially is part of a system > > of systems. > > > > To punctuate your example, I also like to recall the classic EIP > > banking/loan example [3] > > > > [3] http://www.eaipatterns.com/ComposedMessagingExample.html > > > > On Fri, Nov 1, 2013 at 12:48 PM, Chris Geer > wrote: > > > On Fri, Nov 1, 2013 at 6:56 AM, Danny Sullivan >wrote: > > > > > >> Perhaps it would be best for me to take a step back and ask what > problem > > >> are we trying to solve by using Camel? My understanding is that it is > to > > >> allow distribution of Streams to multiple servers. It would be > helpful to > > >> me to see another open source application that has this deployment > > >> structure (it doesn't have to use Camel, just an application that has > > >> multiple jars communicating with one another). Does anyone have any > > >> suggestions of software for me to look at? > > >> > > > > > > Danny, I wish I could just show you my system we are building. > Honestly I > > > think you are looking at this the wrong way. You are looking at it > from a > > > software development perspective (jars, protocols) and not from a > > > business/system perspective (messages, components). So let me use an > > > example. > > > > > > Let's say I have a business that takes order for widgets. I want the > order > > > taking to be streamlined for the user. To make that happen my order > entry > > > system does some basic checks when the user is placing the order and if > > > those pass it saves the order to the database. To kick off the other > > > processing, whenever I save an order to the database I sent a message > to a > > > Topic (pub/sub) saying either an order was created or an order was > updated. > > > Now at this point there is a lot more work I have to do to get the > order > > > shipped to the customer but it's an async action to the order taking. > At > > > this point the order has been placed and the customer can go about > their > > > business. > > > > > > I also have a component I wrote that listens to the Topic for "Order > > > Created" messages. It takes each message, retrieves the order does some > > > inventory processing to get the order ready to ship. At this point the > > > order is complete and shipped to the customer. > > > > > > Now, in this simple example you could argue that the order entry system > > > could just call the order processing component directly but that would > > > preclude some extensibility in the future. > > > > > > A few months down the road I get a requirement to be able to notify the > > > sales team whenever an order is placed by a certain set of customers. > In a > > > traditional model I'd have to modify the order entry component and put > in > > > logic to handle that. In this architecture all I have to do is build a > > > different component that listens to that same Topic for order by > certain > > > customers. When it gets a notification it can then do the logic to > send the > > > message. The beauty of this is the order entry system doesn't' have to > be > > > modified and there is no risk of inserting a bug into that core > process. > > > > > > So you ask, where does camel come in here. It could be used in lots of > > > places but the natural fit is for the shipment processing component to > be a > > > camel route that listens to the Topic and calls a java object when > messages > > > come in. Also for the notification components, the whole thing could > > > probably be a camel route (0 Java code). > > > > > > This is a pretty simple example and there are lots of other benefits I > > > didn't talk about like redundancy and deployment flexibility. > > > > > > Does that help? > > > > > >> > > >> > Date: Thu, 31 Oct 2013 20:23:26 -0400 > > >> > Subject: Re: [DISCUSS] Switching Streams from Camel deployment to > .war > > >> deployment > > >> > From: jletourneau80@gmail.com > > >> > To: dev@streams.incubator.apache.org > > >> > > > >> > So Danny in case it was lost in the ensuing transport discussion, > > >> > Camel abstracts the transport, so you don't have to care if its http > > >> > or whatever, only about the message format. This is helpful if you > > >> > care about plugging in components that DO care about the transport > of > > >> > course. To answer your other questions, the downside to having the > > >> > components all work via http (and not using messaging/Camel) is that > > >> > everything immediately becomes point to point, which isn't a > > >> > performance issue in its own right but limits scaling to more than > one > > >> > component being at the receiving end/fulfillment for instance as > we've > > >> > been discussing as being optimal. > > >> > > > >> > On Thu, Oct 31, 2013 at 5:38 PM, Matt Franklin < > m.ben.franklin@gmail.com> > > >> wrote: > > >> > > On Thu, Oct 31, 2013 at 5:34 PM, Chris Geer < > chris@cxtsoftware.com> > > >> wrote: > > >> > > > > >> > >> On Thu, Oct 31, 2013 at 2:32 PM, Matt Franklin < > > >> m.ben.franklin@gmail.com > > >> > >> >wrote: > > >> > >> > > >> > >> > On Thu, Oct 31, 2013 at 5:28 PM, Chris Geer < > chris@cxtsoftware.com> > > >> > >> wrote: > > >> > >> > > > >> > >> > > On Thu, Oct 31, 2013 at 2:23 PM, Matt Franklin < > > >> > >> m.ben.franklin@gmail.com > > >> > >> > > >wrote: > > >> > >> > > > > >> > >> > > > On Thu, Oct 31, 2013 at 5:04 PM, Danny Sullivan < > > >> > >> > dsullivan7@hotmail.com > > >> > >> > > > >wrote: > > >> > >> > > > > > >> > >> > > > > I'm not quite following this, so I apologize. What I'm > trying > > >> to do > > >> > >> > is > > >> > >> > > > > programmatically make a request to a jar running on a > > >> separate jvm > > >> > >> > and > > >> > >> > > > get > > >> > >> > > > > the response from that call all within the same method. > > >> Similar to > > >> > >> > this > > >> > >> > > > > http request: > > >> > >> > > > > > > >> > >> > > > > HttpGet httpget = new HttpGet(); > > >> > >> > > > > > > >> > >> > > > > httpget.setURI(new URI("www.streams-persistence.com")); > > >> > >> > > > > > > >> > >> > > > > CloseableHttpResponse response = > httpclient.execute(httpget); > > >> > >> > > > > > > >> > >> > > > > //do stuff with the response... > > >> > >> > > > > > > >> > >> > > > > > > >> > >> > > > > > > >> > >> > > > > > > >> > >> > > > > I imagine that this would translate to something in Camel > > >> similar > > >> > >> to > > >> > >> > > > this: > > >> > >> > > > > > > >> > >> > > > > > > >> > >> > > > > > > >> > >> > > > > > > >> > >> > > > > uri="bean:subscriberService?method=getAllSubscribers"/> > > >> > >> > > > > > > >> > >> > > > > > >> > >> > > > > > > >> > >> > > > > > >> > >> > > > > >> > >> > > > >> > >> > > >> > uri="addressToStreamsPersistenceJar?method=selectAllSubscribersFromDatabase"/> > > >> > >> > > > > > > >> > >> > > > > > > >> > >> > > > > > > >> > >> > > > > > >> > >> > > > Camel uses AMQP as a messaging system so that you don't > have to > > >> > >> create > > >> > >> > > HTTP > > >> > >> > > > requests between sources, though it is possible to do so. > In > > >> Storm, > > >> > >> we > > >> > >> > > > usually try to use a buffer system like Kafka to do the > same. > > >> > >> > > > > > >> > >> > > > > >> > >> > > Camel can use AMQP through ActiveMQ. > > >> > >> > > > > >> > >> > > > > >> > >> > Can being the operative word. I haven't done much with Camel. > What > > >> > >> other > > >> > >> > transports are available. > > >> > >> > > > >> > >> > OpenWire (native ActiveMQ) > > >> > >> AMQP > > >> > >> STOMP > > >> > >> HTTP (can be a server or client) > > >> > >> Mail > > >> > >> Amazon SQS > > >> > >> XMPP > > >> > >> ....hundred or so more > > >> > >> > > >> > >> http://camel.apache.org/components.html > > >> > > > > >> > > > > >> > > > > >> > > Nice. I was expecting RTFM, so thanks for catering to my > laziness. > > >> > > > > >> > > > > >> > >> > > >> > >> > > >> > >> > > > >> > >> > > > > > >> > >> > > > > > >> > >> > > > > > > >> > >> > > > > > > >> > >> > > > > But it is unclear what the actual implementation would > be. > > >> > >> > > > > > > >> > >> > > > > > > >> > >> > > > > This actually brings me to another suggestion. Would > there be > > >> a big > > >> > >> > > > > performance impact to have communication between the > software > > >> > >> > > components > > >> > >> > > > > occur between http? Say the 5 software components I > outlined > > >> > >> earlier > > >> > >> > > were > > >> > >> > > > > packaged as 5 separate wars. These wars could communicate > > >> with each > > >> > >> > > other > > >> > >> > > > > via get a post requests. This sounds unconventional > offhand > > >> so I'd > > >> > >> > like > > >> > >> > > > to > > >> > >> > > > > hear some thoughts on it. > > >> > >> > > > > > > >> > >> > > > > > >> > >> > > > It is most certainly possible, but IMO probably not the > best > > >> option > > >> > >> for > > >> > >> > > > success. Protocols like PubSubHubbub use HTTP for a > message > > >> > >> transport. > > >> > >> > > > > > >> > >> > > > > > >> > >> > > > > -Danny > > >> > >> > > > > > > >> > >> > > > > > Date: Thu, 31 Oct 2013 13:28:41 -0700 > > >> > >> > > > > > Subject: Re: [DISCUSS] Switching Streams from Camel > > >> deployment to > > >> > >> > > .war > > >> > >> > > > > deployment > > >> > >> > > > > > From: chris@cxtsoftware.com > > >> > >> > > > > > To: dev@streams.incubator.apache.org > > >> > >> > > > > > > > >> > >> > > > > > Or Content Enricher [2] > > >> > >> > > > > > > > >> > >> > > > > > [2] http://camel.apache.org/content-enricher.html > > >> > >> > > > > > > > >> > >> > > > > > > > >> > >> > > > > > On Thu, Oct 31, 2013 at 12:37 PM, Jason Letourneau > > >> > >> > > > > > wrote: > > >> > >> > > > > > > > >> > >> > > > > > > check out the link here[1] > > >> > >> > > > > > > > > >> > >> > > > > > > > > >> > >> > > > > > > [1]http://camel.apache.org/request-reply.html > > >> > >> > > > > > > > > >> > >> > > > > > > On Thu, Oct 31, 2013 at 2:32 PM, Danny Sullivan < > > >> > >> > > > > dsullivan7@hotmail.com> > > >> > >> > > > > > > wrote: > > >> > >> > > > > > > > I have a quick Camel question that I arrived at in > the > > >> > >> > > > > implementation of > > >> > >> > > > > > > these new components: > > >> > >> > > > > > > > > > >> > >> > > > > > > > Lets say I have a method in streams-activity.jar > that > > >> needs > > >> > >> all > > >> > >> > > > > > > subscribers in the database. This would require a > call to > > >> the > > >> > >> > > > > > > streams-persistence.jar. So far, I've seen camel used > > >> mostly > > >> > >> for > > >> > >> > > > > passing > > >> > >> > > > > > > data through the application, but not for making a > single > > >> > >> > > > > request-reponse > > >> > >> > > > > > > from within a method. How can I use Camel to get a > list > > >> of all > > >> > >> > > > > subscribers > > >> > >> > > > > > > in the streams-persistence.jar from the > > >> streams-activity.jar? > > >> > >> > > > > > > > > > >> > >> > > > > > > >> From: dsullivan7@hotmail.com > > >> > >> > > > > > > >> To: dev@streams.incubator.apache.org > > >> > >> > > > > > > >> Subject: RE: [DISCUSS] Switching Streams from > Camel > > >> > >> deployment > > >> > >> > > to > > >> > >> > > > > .war > > >> > >> > > > > > > deployment > > >> > >> > > > > > > >> Date: Thu, 31 Oct 2013 08:40:44 -0400 > > >> > >> > > > > > > >> > > >> > >> > > > > > > >> Excellent, I'll write up something as a proof of > > >> concept and > > >> > >> > we > > >> > >> > > > can > > >> > >> > > > > > > discuss further to make sure everything is vanilla. > > >> > >> > > > > > > >> > > >> > >> > > > > > > >> > Date: Wed, 30 Oct 2013 17:01:28 -0400 > > >> > >> > > > > > > >> > Subject: Re: [DISCUSS] Switching Streams from > Camel > > >> > >> > deployment > > >> > >> > > > to > > >> > >> > > > > > > .war deployment > > >> > >> > > > > > > >> > From: jletourneau80@gmail.com > > >> > >> > > > > > > >> > To: dev@streams.incubator.apache.org > > >> > >> > > > > > > >> > > > >> > >> > > > > > > >> > That sounds pretty promising to me. > > >> > >> > > > > > > >> > > > >> > >> > > > > > > >> > On Wed, Oct 30, 2013 at 2:38 PM, Danny Sullivan > < > > >> > >> > > > > > > dsullivan7@hotmail.com> wrote: > > >> > >> > > > > > > >> > > Thanks for the feedback. You have an > interesting > > >> point > > >> > >> > about > > >> > >> > > > the > > >> > >> > > > > > > url linking to a separate processing space. Let me > tie my > > >> > >> answer > > >> > >> > > into > > >> > >> > > > > your > > >> > >> > > > > > > last question about "advocating for the simplicity at > > >> > >> > registration > > >> > >> > > to > > >> > >> > > > > give > > >> > >> > > > > > > up flexibility at registration, but retaining the > inner > > >> "guts" > > >> > >> of > > >> > >> > > > > > > EIP/messaging". Consider a new architecture: > > >> > >> > > > > > > >> > > > > >> > >> > > > > > > >> > > streams-web.war: single entry point to > > >> application, but > > >> > >> > > > > functions > > >> > >> > > > > > > ONLY as an entry point. From here Camel routes the > > >> incoming > > >> > >> > > requests > > >> > >> > > > > to 4 > > >> > >> > > > > > > separate jarssubscriber-registration.jar: subscriber > > >> > >> registration > > >> > >> > > > > > > publisher-registration.jar: publisher > > >> registrationactivity.jar: > > >> > >> > > > returns > > >> > >> > > > > > > activity (also contains subscriber warehouse and > storm > > >> activity > > >> > >> > > > > > > aggregator)publish.jar: publishes > > >> > >> activitystreams-cassandra.jar: > > >> > >> > > the > > >> > >> > > > > above > > >> > >> > > > > > > 4 jars would all have a hook into this jar which > would > > >> function > > >> > >> > as > > >> > >> > > a > > >> > >> > > > > hook > > >> > >> > > > > > > onto the database. Each jar would have camel route > output > > >> to > > >> > >> this > > >> > >> > > > jar. > > >> > >> > > > > > > >> > > > > >> > >> > > > > > > >> > > In this implementation, Camel would no longer > be > > >> the > > >> > >> entry > > >> > >> > > and > > >> > >> > > > > exit > > >> > >> > > > > > > point of a client to the application, but would > handle the > > >> > >> > > > > communication > > >> > >> > > > > > > between components. The flow of activity through the > > >> > >> application > > >> > >> > > > would > > >> > >> > > > > be > > >> > >> > > > > > > method based in each jar. This would allow > deployment on > > >> up to > > >> > >> 6 > > >> > >> > > > > different > > >> > >> > > > > > > process spaces. However, this does not address that > there > > >> is a > > >> > >> > > single > > >> > >> > > > > > > server entry point, but I'm not sure if it was a > concern > > >> in the > > >> > >> > > first > > >> > >> > > > > place. > > >> > >> > > > > > > >> > > > > >> > >> > > > > > > >> > > My argument, at its basis, is that we should > move > > >> away > > >> > >> > from > > >> > >> > > > > using > > >> > >> > > > > > > Camel as the entry point to the application. I would > be > > >> happy > > >> > >> to > > >> > >> > > > > maintain > > >> > >> > > > > > > messaging between components. > > >> > >> > > > > > > >> > > > > >> > >> > > > > > > >> > >> Date: Wed, 30 Oct 2013 12:32:55 -0400 > > >> > >> > > > > > > >> > >> Subject: Re: [DISCUSS] Switching Streams from > > >> Camel > > >> > >> > > > deployment > > >> > >> > > > > to > > >> > >> > > > > > > .war deployment > > >> > >> > > > > > > >> > >> From: jletourneau80@gmail.com > > >> > >> > > > > > > >> > >> To: dev@streams.incubator.apache.org > > >> > >> > > > > > > >> > >> > > >> > >> > > > > > > >> > >> An interesting use case that I am holding > onto is > > >> the > > >> > >> > > ability > > >> > >> > > > > for > > >> > >> > > > > > > >> > >> publishers to register via a single URL > > >> (registration > > >> > >> > > > > endpoint), > > >> > >> > > > > > > but > > >> > >> > > > > > > >> > >> be sent a URL back to post to a different > process > > >> space > > >> > >> > for > > >> > >> > > > > actual > > >> > >> > > > > > > >> > >> publishing. The same is true on the > subscriber > > >> front. > > >> > >> > > > > Currently, > > >> > >> > > > > > > the > > >> > >> > > > > > > >> > >> Camel/EIP infrastructure abstracts this > because > > >> > >> different > > >> > >> > > > > > > components > > >> > >> > > > > > > >> > >> deployed in different process spaces > handling the > > >> route > > >> > >> > > > > creation > > >> > >> > > > > > > can > > >> > >> > > > > > > >> > >> just be bolted onto a running Streams > instance > > >> without > > >> > >> > new > > >> > >> > > > > > > subs/pubs > > >> > >> > > > > > > >> > >> behaving any differently than existing. This > > >> seems to > > >> > >> > be a > > >> > >> > > > > > > >> > >> potentially critical scaling point. Is > there a > > >> way to > > >> > >> do > > >> > >> > > > this > > >> > >> > > > > with > > >> > >> > > > > > > >> > >> the Spring solution? > > >> > >> > > > > > > >> > >> > > >> > >> > > > > > > >> > >> The persistence point is a good one, though I > > >> would > > >> > >> > > classify > > >> > >> > > > > that > > >> > >> > > > > > > as > > >> > >> > > > > > > >> > >> "not implemented" vs "not possible" (not > that you > > >> > >> were). > > >> > >> > > > > > > >> > >> > > >> > >> > > > > > > >> > >> I'm not married to Camel, I just like the EIP > > >> approach > > >> > >> to > > >> > >> > > > > building > > >> > >> > > > > > > >> > >> something that is ultimately a messaging > system. > > >> There > > >> > >> > are > > >> > >> > > > > known > > >> > >> > > > > > > >> > >> patterns that solve at least a subset of the > > >> problems > > >> > >> > > Streams > > >> > >> > > > > is > > >> > >> > > > > > > >> > >> trying to solve and implementations that can > > >> handle the > > >> > >> > > load > > >> > >> > > > > and > > >> > >> > > > > > > I'll > > >> > >> > > > > > > >> > >> reiterate flexibility == complexity almost > always. > > >> > >> > > > > > > >> > >> > > >> > >> > > > > > > >> > >> It comes right back to the central question: > Do > > >> you > > >> > >> want > > >> > >> > > > > > > flexibility > > >> > >> > > > > > > >> > >> or simplicity? It doesn't have to be black > and > > >> white > > >> > >> > > either > > >> > >> > > > I > > >> > >> > > > > > > don't > > >> > >> > > > > > > >> > >> think... > > >> > >> > > > > > > >> > >> > > >> > >> > > > > > > >> > >> More pointedly: Where should we give up > > >> flexibility for > > >> > >> > > > > > > simplicity? I > > >> > >> > > > > > > >> > >> read that Danny is advocating for the > simplicity > > >> at > > >> > >> > > > > registration to > > >> > >> > > > > > > >> > >> give up flexibility at registration, but > > >> retaining the > > >> > >> > > inner > > >> > >> > > > > > > "guts" of > > >> > >> > > > > > > >> > >> EIP/messaging? Thoughts? > > >> > >> > > > > > > >> > >> > > >> > >> > > > > > > >> > >> On Wed, Oct 30, 2013 at 11:37 AM, Danny > Sullivan < > > >> > >> > > > > > > dsullivan7@hotmail.com> wrote: > > >> > >> > > > > > > >> > >> > My argument is not for the IoC pattern as > that > > >> can be > > >> > >> > > (and > > >> > >> > > > > has > > >> > >> > > > > > > been) implemented alongside Camel. My main argument > is > > >> that the > > >> > >> > > > syntax > > >> > >> > > > > at > > >> > >> > > > > > > the entry point is not only familiar but much > simpler. > > >> This > > >> > >> > > wouldn't > > >> > >> > > > > be a > > >> > >> > > > > > > very strong argument if the Camel implementation > wasn't > > >> much > > >> > >> more > > >> > >> > > > > > > complicated but I feel that it is the case. Also, > looking > > >> > >> toward > > >> > >> > > the > > >> > >> > > > > > > future, if the server is restarted, in-routes are > lost in > > >> > >> Camel. > > >> > >> > > The > > >> > >> > > > > way to > > >> > >> > > > > > > curb this is to persist the dynamic routes that Camel > > >> creates, > > >> > >> > and > > >> > >> > > > > then on > > >> > >> > > > > > > start up pull every one of these routes and recreate > a > > >> dynamic > > >> > >> > > route > > >> > >> > > > > for > > >> > >> > > > > > > each one. Not only is this much easier to implement > using > > >> the > > >> > >> > > Spring > > >> > >> > > > > web > > >> > >> > > > > > > implementation, but it already has been implemented > and > > >> you can > > >> > >> > try > > >> > >> > > > it > > >> > >> > > > > by > > >> > >> > > > > > > checking out the webservice branch, registering a > > >> subscriber, > > >> > >> > > > > restarting > > >> > >> > > > > > > tomcat, and using the same url you had before. This > will > > >> allow > > >> > >> > > > > subscribers > > >> > >> > > > > > > to hang on to their urls once they register. (the > same is > > >> true > > >> > >> > for > > >> > >> > > > > > > publishers: you can post via the same url after > restarting > > >> > >> > tomcat) > > >> > >> > > > > > > >> > >> > > > >> > >> > > > > > > >> > >> >> Date: Wed, 30 Oct 2013 11:00:21 -0400 > > >> > >> > > > > > > >> > >> >> Subject: Re: [DISCUSS] Switching Streams > from > > >> Camel > > >> > >> > > > > deployment > > >> > >> > > > > > > to .war deployment > > >> > >> > > > > > > >> > >> >> From: jletourneau80@gmail.com > > >> > >> > > > > > > >> > >> >> To: dev@streams.incubator.apache.org > > >> > >> > > > > > > >> > >> >> > > >> > >> > > > > > > >> > >> >> To be fair, while the current > implementation is > > >> > >> > heavily > > >> > >> > > > > > > camel-based, > > >> > >> > > > > > > >> > >> >> all of the interfaces related to Streams > > >> > >> functionality > > >> > >> > > are > > >> > >> > > > > not. > > >> > >> > > > > > > The > > >> > >> > > > > > > >> > >> >> current model maps to what Matt has > outlined > > >> in my > > >> > >> > > > opinion, > > >> > >> > > > > > > though > > >> > >> > > > > > > >> > >> >> packing names etc. probably don't follow > that > > >> exact > > >> > >> > > > pattern. > > >> > >> > > > > > > >> > >> >> > > >> > >> > > > > > > >> > >> >> With regards to the complexity and > different > > >> > >> > components > > >> > >> > > in > > >> > >> > > > > the > > >> > >> > > > > > > >> > >> >> registration process, this was a cut at > the > > >> > >> > abstraction > > >> > >> > > > > based > > >> > >> > > > > > > on the > > >> > >> > > > > > > >> > >> >> assumption that different implementations > may > > >> be > > >> > >> > plugged > > >> > >> > > > in > > >> > >> > > > > and > > >> > >> > > > > > > in > > >> > >> > > > > > > >> > >> >> fact may live on different processor space > > >> (ie. a > > >> > >> > > polling > > >> > >> > > > > > > publisher vs > > >> > >> > > > > > > >> > >> >> a push publisher may be instantiated on > > >> different > > >> > >> > > servers > > >> > >> > > > > but > > >> > >> > > > > > > the > > >> > >> > > > > > > >> > >> >> registration URL is staticly defined). > > >> > >> > > > > > > >> > >> >> > > >> > >> > > > > > > >> > >> >> Is the main argument I am seeing for > Spring > > >> the > > >> > >> > > > > familiarity of > > >> > >> > > > > > > its > > >> > >> > > > > > > >> > >> >> IoC pattern implementation and syntax at > the > > >> entry > > >> > >> > > point? > > >> > >> > > > > > > >> > >> >> > > >> > >> > > > > > > >> > >> >> On Wed, Oct 30, 2013 at 10:53 AM, Danny > > >> Sullivan < > > >> > >> > > > > > > dsullivan7@hotmail.com> wrote: > > >> > >> > > > > > > >> > >> >> > Could you clarify whether the same entry > > >> points > > >> > >> > would > > >> > >> > > > > exist > > >> > >> > > > > > > for the camel implementation of the core > (implementing the > > >> > >> > > "process" > > >> > >> > > > > > > method/ using a DynammicRouteBuilder) or would the > > >> webservice > > >> > >> be > > >> > >> > > the > > >> > >> > > > > sole > > >> > >> > > > > > > entry point to Streams and after it enters would it > hand > > >> it off > > >> > >> > to > > >> > >> > > > > Camel? > > >> > >> > > > > > > And what would be the entry point for the Storm > > >> implementation? > > >> > >> > > > > > > >> > >> >> > -Danny > > >> > >> > > > > > > >> > >> >> > > > >> > >> > > > > > > >> > >> >> >> Date: Wed, 30 Oct 2013 10:13:04 -0400 > > >> > >> > > > > > > >> > >> >> >> Subject: Re: [DISCUSS] Switching > Streams > > >> from > > >> > >> Camel > > >> > >> > > > > > > deployment to .war deployment > > >> > >> > > > > > > >> > >> >> >> From: m.ben.franklin@gmail.com > > >> > >> > > > > > > >> > >> >> >> To: dev@streams.incubator.apache.org > > >> > >> > > > > > > >> > >> >> >> > > >> > >> > > > > > > >> > >> >> >> On Tue, Oct 29, 2013 at 2:55 PM, Danny > > >> Sullivan < > > >> > >> > > > > > > dsullivan7@hotmail.com>wrote: > > >> > >> > > > > > > >> > >> >> >> > > >> > >> > > > > > > >> > >> >> >> > My case for switching from OSGi is > for > > >> > >> simplicity > > >> > >> > > in > > >> > >> > > > > > > design. To follow the > > >> > >> > > > > > > >> > >> >> >> > path of an activity through Streams > in the > > >> > >> > > > webservice, > > >> > >> > > > > > > there is one main > > >> > >> > > > > > > >> > >> >> >> > things the developer needs to > understand: > > >> > >> > > > > > > >> > >> >> >> > The @RequestMapping annotation > specifies > > >> the > > >> > >> HTTP > > >> > >> > > > entry > > >> > >> > > > > > > point > > >> > >> > > > > > > >> > >> >> >> > There are 4 @RequestMapping > annotations > > >> that > > >> > >> > > > > correspond to > > >> > >> > > > > > > each of the 4 > > >> > >> > > > > > > >> > >> >> >> > ways a user enters the application: > > >> > >> registering a > > >> > >> > > > > > > publisher, registering a > > >> > >> > > > > > > >> > >> >> >> > subscriber, publishing activity, and > > >> getting an > > >> > >> > > > > activity > > >> > >> > > > > > > stream. Where > > >> > >> > > > > > > >> > >> >> >> > these are located in the source code > can > > >> be > > >> > >> found > > >> > >> > > by > > >> > >> > > > > > > searching for the > > >> > >> > > > > > > >> > >> >> >> > paths specified in the documentation > > >> (search > > >> > >> for > > >> > >> > > > > > > "/publisherRegister", > > >> > >> > > > > > > >> > >> >> >> > "/subscriberRegister", > "/publishActivity", > > >> > >> > > > > "/getActivity" > > >> > >> > > > > > > which will all > > >> > >> > > > > > > >> > >> >> >> > lead you to > StreamsWebController.java). > > >> From > > >> > >> the > > >> > >> > > > > methods > > >> > >> > > > > > > that process > > >> > >> > > > > > > >> > >> >> >> > requests, the flow through the > > >> application is > > >> > >> > > through > > >> > >> > > > > > > methods which can be > > >> > >> > > > > > > >> > >> >> >> > understood by most Java programmers. > > >> > >> > > > > > > >> > >> >> >> > The flow of activities through the > current > > >> > >> trunk > > >> > >> > > > > branch is > > >> > >> > > > > > > understood as > > >> > >> > > > > > > >> > >> >> >> > follows: > > >> > >> > > > > > > >> > >> >> >> > The string "/publisher/register" (the > > >> entry > > >> > >> point > > >> > >> > > to > > >> > >> > > > > > > register a publisher > > >> > >> > > > > > > >> > >> >> >> > specified in the documentation) is > the > > >> value of > > >> > >> > the > > >> > >> > > > > > > >> > >> >> >> > consumer.registrationEndpoint > property > > >> defined > > >> > >> in > > >> > >> > > > > > > streams.propertiesThe > > >> > >> > > > > > > >> > >> >> >> > camelContext.xml specifies an > endpoint > > >> with the > > >> > >> > id > > >> > >> > > > > > > >> > >> >> >> > consumerRegistrationEndpoint to have > a uri > > >> > >> equal > > >> > >> > to > > >> > >> > > > the > > >> > >> > > > > > > propertyThe > > >> > >> > > > > > > >> > >> >> >> > consumerRegistrationEndpoint routes > from > > >> uri > > >> > >> > > > > > > direct:publisher register with > > >> > >> > > > > > > >> > >> >> >> > the bean > activityRegistrationProcessor > > >> nested > > >> > >> in > > >> > >> > > > > between > > >> > >> > > > > > > the routeThe > > >> > >> > > > > > > >> > >> >> >> > streams-eip-applicationContext > contains a > > >> bean > > >> > >> > with > > >> > >> > > > > the id > > >> > >> > > > > > > >> > >> >> >> > activityRegistrationProcessor > created for > > >> the > > >> > >> > class > > >> > >> > > > > > > >> > >> >> >> > > ActivityPublisherRegistrationProcessorThe > > >> > >> > exchange > > >> > >> > > > will > > >> > >> > > > > > > enter the "process" > > >> > >> > > > > > > >> > >> >> >> > method of the class > > >> > >> > > > > > > ActivityPublisherRegisitrationProcessor and that this > > >> > >> > > > > > > >> > >> >> >> > is because this class implements the > > >> > >> "Processor" > > >> > >> > > > > interface > > >> > >> > > > > > > provided by > > >> > >> > > > > > > >> > >> >> >> > CamelThe direct:add-publisher-route > takes > > >> the > > >> > >> > > > exchange > > >> > >> > > > > > > output from the > > >> > >> > > > > > > >> > >> >> >> > "process" method and routes it to the > > >> > >> > > > > > > activityRegistrationProcessor > > >> > >> > > > > > > >> > >> >> >> > "register" method. The bean > > >> > >> > > > > activityRegistrationProcessor > > >> > >> > > > > > > is defined in the > > >> > >> > > > > > > >> > >> >> >> > > streams-eip-osgi-component-import.xmlThe > > >> output > > >> > >> > > from > > >> > >> > > > > this > > >> > >> > > > > > > method is then > > >> > >> > > > > > > >> > >> >> >> > sent to the > "createNewRouteForConsumer" > > >> method > > >> > >> of > > >> > >> > > > > > > activityConsumerRouter. > > >> > >> > > > > > > >> > >> >> >> > This method creates a new route for > the > > >> newly > > >> > >> > > > > registered > > >> > >> > > > > > > publisher using > > >> > >> > > > > > > >> > >> >> >> > the private static final class > > >> > >> > > > > DynamicConsumerRouteBuilder > > >> > >> > > > > > > which is > > >> > >> > > > > > > >> > >> >> >> > required to extend RouteBuilder > which is > > >> > >> provided > > >> > >> > > by > > >> > >> > > > > > > Camel. This > > >> > >> > > > > > > >> > >> >> >> > DynamicConsumerRouteBuilder contains > > >> several > > >> > >> > > methods: > > >> > >> > > > > > > >> > >> >> >> > getConsumerReceiveMethod() (which > > >> corresponds > > >> > >> to > > >> > >> > > > @Value > > >> > >> > > > > > > >> > >> >> >> > ${consumer.receiveMethod} which > > >> corresponds to > > >> > >> > > > > "receive"), > > >> > >> > > > > > > >> > >> >> >> > getConsumerSplitMethod() (@Value > > >> > >> > > > > > > ${consumer.splitMethod},"split"), and > > >> > >> > > > > > > >> > >> >> >> > getConsumerActivityQUri() (@Value > > >> > >> > > > > ${consumerActivityQUri}, > > >> > >> > > > > > > >> > >> >> >> > "direct:activityQ"). This is > different > > >> than the > > >> > >> > > > > > > camelContext.xml in that > > >> > >> > > > > > > >> > >> >> >> > the route is being created > > >> programmatically. > > >> > >> What > > >> > >> > > > this > > >> > >> > > > > is > > >> > >> > > > > > > doing is routing > > >> > >> > > > > > > >> > >> >> >> > input from the inroute url (which > Camel > > >> does > > >> > >> > > > > automatically > > >> > >> > > > > > > through the > > >> > >> > > > > > > >> > >> >> >> > configure method which is required > to be > > >> > >> > > overridden), > > >> > >> > > > > to > > >> > >> > > > > > > the "receive" > > >> > >> > > > > > > >> > >> >> >> > method of ActivityConsumer, then to > the > > >> "split" > > >> > >> > > > method > > >> > >> > > > > of > > >> > >> > > > > > > ActivityConsumer, > > >> > >> > > > > > > >> > >> >> >> > and then to the "direct:activityQ" > which > > >> if you > > >> > >> > > look > > >> > >> > > > > back > > >> > >> > > > > > > in the > > >> > >> > > > > > > >> > >> >> >> > camelContext.xml routes to the > > >> > >> > > > > "activemq:queue:activities" > > >> > >> > > > > > > which then > > >> > >> > > > > > > >> > >> >> >> > routes to "receiveExchange" > > >> > >> > > > > > > >> > >> >> >> > This is the process to register a > > >> publisher. > > >> > >> The > > >> > >> > > > > process > > >> > >> > > > > > > for registering a > > >> > >> > > > > > > >> > >> >> >> > subscriber is relatively the same > though > > >> it > > >> > >> > > involves > > >> > >> > > > > > > separate classes with > > >> > >> > > > > > > >> > >> >> >> > their own private static final > > >> RouteBuilder > > >> > >> > class. > > >> > >> > > > > From my > > >> > >> > > > > > > perspective, the > > >> > >> > > > > > > >> > >> >> >> > two most difficult things with > setting > > >> this > > >> > >> > project > > >> > >> > > > up > > >> > >> > > > > > > were understanding > > >> > >> > > > > > > >> > >> >> >> > that the "process" method of the > class > > >> that > > >> > >> > > > implements > > >> > >> > > > > > > "Processor" is the > > >> > >> > > > > > > >> > >> >> >> > entry point and the > > >> DynamicConsumerRouteBuilder > > >> > >> > > > creates > > >> > >> > > > > > > the second entry > > >> > >> > > > > > > >> > >> >> >> > point (The 5th and last points). This > > >> made the > > >> > >> > > > project > > >> > >> > > > > > > very, VERY hard to > > >> > >> > > > > > > >> > >> >> >> > understand. > > >> > >> > > > > > > >> > >> >> >> > In addition to simplicity of design, > the > > >> mvn > > >> > >> > clean > > >> > >> > > > > install > > >> > >> > > > > > > of the web > > >> > >> > > > > > > >> > >> >> >> > service project is much faster and > small > > >> scale > > >> > >> > > > activity > > >> > >> > > > > > > publishing is also > > >> > >> > > > > > > >> > >> >> >> > faster (see my email about load > testing). > > >> These > > >> > >> > are > > >> > >> > > > > minor > > >> > >> > > > > > > points though as > > >> > >> > > > > > > >> > >> >> >> > compilation has no effect on > deployment. > > >> OSGi > > >> > >> > does > > >> > >> > > > add > > >> > >> > > > > the > > >> > >> > > > > > > benefit of > > >> > >> > > > > > > >> > >> >> >> > modularized programming which is > valuable, > > >> > >> > though I > > >> > >> > > > > think > > >> > >> > > > > > > the added > > >> > >> > > > > > > >> > >> >> >> > complexity of Camel merits moving the > > >> project > > >> > >> > away > > >> > >> > > > from > > >> > >> > > > > > > this paradigm. > > >> > >> > > > > > > >> > >> >> >> > > > >> > >> > > > > > > >> > >> >> >> > > >> > >> > > > > > > >> > >> >> >> I agree that the project is pretty > > >> difficult to > > >> > >> > > > > understand > > >> > >> > > > > > > ATM. I think > > >> > >> > > > > > > >> > >> >> >> what we need to do is think about what > the > > >> > >> > > > > responsibilities > > >> > >> > > > > > > of the code are > > >> > >> > > > > > > >> > >> >> >> and allow for different implementations > > >> that are > > >> > >> > not > > >> > >> > > so > > >> > >> > > > > > > tightly coupled as > > >> > >> > > > > > > >> > >> >> >> they are now. For instance, having > worked > > >> with > > >> > >> > Storm > > >> > >> > > > to > > >> > >> > > > > > > ingest millions of > > >> > >> > > > > > > >> > >> >> >> activities a day, I personally would > like > > >> to see > > >> > >> > > > streams > > >> > >> > > > > be > > >> > >> > > > > > > responsible for > > >> > >> > > > > > > >> > >> >> >> defining an over-arching orchestration > > >> model that > > >> > >> > can > > >> > >> > > > be > > >> > >> > > > > > > implemented within > > >> > >> > > > > > > >> > >> >> >> a single war or on top of a distributed > > >> system. > > >> > >> > This > > >> > >> > > > > would > > >> > >> > > > > > > look something > > >> > >> > > > > > > >> > >> >> >> like the follows: > > >> > >> > > > > > > >> > >> >> >> > > >> > >> > > > > > > >> > >> >> >> |__ Streams-Core (Base classes, > interfaces, > > >> > >> > > utilities, > > >> > >> > > > > > > extensions, etc) > > >> > >> > > > > > > >> > >> >> >> | > > >> > >> > > > > > > >> > >> >> >> |__ Streams-Storm (Storm > implementation of > > >> the > > >> > >> > core) > > >> > >> > > > > > > >> > >> >> >> | > > >> > >> > > > > > > >> > >> >> >> |__ Streams-Camel (Camel > implementation of > > >> the > > >> > >> > core) > > >> > >> > > > > > > >> > >> >> >> | > > >> > >> > > > > > > >> > >> >> >> |__ Streams-WS (Web service > implementation) > > >> > >> > > > > > > >> > >> >> >> > > >> > >> > > > > > > >> > >> >> >> > > >> > >> > > > > > > >> > >> >> >> > Danny > > >> > >> > > > > > > >> > >> >> >> > > > >> > >> > > > > > > >> > >> >> >> > > > >> > >> > > > > > > >> > >> >> >> > > From: dsullivan7@hotmail.com > > >> > >> > > > > > > >> > >> >> >> > > To: > dev@streams.incubator.apache.org > > >> > >> > > > > > > >> > >> >> >> > > Subject: [DISCUSS] Switching > Streams > > >> from > > >> > >> Camel > > >> > >> > > > > > > deployment to .war > > >> > >> > > > > > > >> > >> >> >> > deployment > > >> > >> > > > > > > >> > >> >> >> > > Date: Tue, 29 Oct 2013 11:07:39 > -0400 > > >> > >> > > > > > > >> > >> >> >> > > > > >> > >> > > > > > > >> > >> >> >> > > The discussion thread for switching > > >> Streams > > >> > >> > from > > >> > >> > > > > > > Camel/osgi/Servicemix > > >> > >> > > > > > > >> > >> >> >> > to a single .war deployment > > >> > >> > > > > > > >> > >> >> >> > > > >> > >> > > > > > > >> > >> >> >> > > > >> > >> > > > > > > >> > >> >> > > > >> > >> > > > > > > >> > >> > > > >> > >> > > > > > > >> > > > > >> > >> > > > > > > >> > > >> > >> > > > > > > > > > >> > >> > > > > > > > > >> > >> > > > > > > >> > >> > > > > > > >> > >> > > > > > >> > >> > > > > >> > >> > > > >> > >> > > >> > > >> > > --047d7b2e4af246dfcd04ea2bcf3b--