Return-Path: 
 <dev-return-468-apmail-streams-dev-archive=streams.apache.org@streams.incubator.apache.org>
X-Original-To: apmail-streams-dev-archive@minotaur.apache.org
Delivered-To: apmail-streams-dev-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7C0C010D90
	for <apmail-streams-dev-archive@minotaur.apache.org>;
 Thu, 31 Oct 2013 21:04:39 +0000 (UTC)
Received: (qmail 58951 invoked by uid 500); 31 Oct 2013 21:04:39 -0000
Delivered-To: apmail-streams-dev-archive@streams.apache.org
Received: (qmail 58918 invoked by uid 500); 31 Oct 2013 21:04:39 -0000
Mailing-List: contact dev-help@streams.incubator.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:dev-help@streams.incubator.apache.org>
List-Unsubscribe: <mailto:dev-unsubscribe@streams.incubator.apache.org>
List-Post: <mailto:dev@streams.incubator.apache.org>
List-Id: <dev.streams.incubator.apache.org>
Reply-To: dev@streams.incubator.apache.org
Delivered-To: mailing list dev@streams.incubator.apache.org
Received: (qmail 58910 invoked by uid 99); 31 Oct 2013 21:04:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Oct 2013 21:04:39 +0000
X-ASF-Spam-Status: No, hits=2.4 required=5.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of dsullivan7@hotmail.com
 designates 65.54.51.79 as permitted sender)
Received: from [65.54.51.79] (HELO snt0-omc3-s42.snt0.hotmail.com)
 (65.54.51.79)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Oct 2013 21:04:31 +0000
Received: from SNT149-W51 ([65.55.90.136]) by snt0-omc3-s42.snt0.hotmail.com
 with Microsoft SMTPSVC(6.0.3790.4675);
	 Thu, 31 Oct 2013 14:04:10 -0700
X-TMN: [jXe/F1c87gVLXooxjqB0QVp3mRQuZpuB]
X-Originating-Email: [dsullivan7@hotmail.com]
Message-ID: <SNT149-W51AB5CB1F1F30F2D3098588A0B0@phx.gbl>
Content-Type: multipart/alternative;
	boundary="_9cb8a643-dbd8-4c37-be26-c644434cb86f_"
From: Danny Sullivan <dsullivan7@hotmail.com>
To: "dev@streams.incubator.apache.org" <dev@streams.incubator.apache.org>
Subject: RE: [DISCUSS] Switching Streams from Camel deployment to .war
 deployment
Date: Thu, 31 Oct 2013 17:04:09 -0400
Importance: Normal
In-Reply-To: 
 <CAFNO4HhB0QbDOYHsbGTL=rPRnyBzActW4gcy3g3N+TxMJT1n4w@mail.gmail.com>
References: 
 <SNT149-W9255D3543C9374C1010DCB8A090@phx.gbl>,<SNT149-W17DBFEA90CE093DF3F90F88A090@phx.gbl>,<CAENpzYWdGg4xK2Ey+CeFqV2jC5DZWyM8ocRX03Yn7o+p9FnoEg@mail.gmail.com>,<SNT149-W478477E9CE78116FB1F0188A0A0@phx.gbl>,<CAL8syObZtmfDBtznouz5k+bEgcG55=W9k8KvKtQBQVNHzNsTBA@mail.gmail.com>,<SNT149-W4688D1837ED8A61183B6278A0A0@phx.gbl>,<CAL8syOadWnYXMOc3X6ooU-c7Wanjo98YiNHaOU2PC9S5+Bou8Q@mail.gmail.com>,<SNT149-W3466D775B64424835F79A38A0A0@phx.gbl>,<CAL8syOYRWHdQvHo_mysSe01vdNAOpvypTe9dBZycJ4J3w=1x+Q@mail.gmail.com>,<SNT149-W6947AB6A21214358E253618A0B0@phx.gbl>,<SNT149-W607E0AEEE5380DED910ABE8A0B0@phx.gbl>,<CAL8syOa2JQ0Lc4ztRxDBo5ecVFLD3kBq15wAcDhVmtMNc+UAcQ@mail.gmail.com>,<CAFNO4HhB0QbDOYHsbGTL=rPRnyBzActW4gcy3g3N+TxMJT1n4w@mail.gmail.com>
MIME-Version: 1.0
X-OriginalArrivalTime: 31 Oct 2013 21:04:10.0225 (UTC)
 FILETIME=[BE981610:01CED67C]
X-Virus-Checked: Checked by ClamAV on apache.org

--_9cb8a643-dbd8-4c37-be26-c644434cb86f_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

I'm not quite following this=2C so I apologize. What I'm trying to do is pr=
ogrammatically make a request to a jar running on a separate jvm and get th=
e response from that call all within the same method. Similar to this http =
request:

HttpGet httpget =3D new HttpGet()=3B

httpget.setURI(new URI("www.streams-persistence.com"))=3B

CloseableHttpResponse response =3D httpclient.execute(httpget)=3B

//do stuff with the response...




I imagine that this would translate to something in Camel similar to this:


<route>

<from uri=3D"bean:subscriberService?method=3DgetAllSubscribers"/>

<inOut uri=3D"addressToStreamsPersistenceJar?method=3DselectAllSubscribersF=
romDatabase"/>

</route>


But it is unclear what the actual implementation would be.


This actually brings me to another suggestion. Would there be a big perform=
ance impact to have communication between the software components occur bet=
ween http? Say the 5 software components I outlined earlier were packaged a=
s 5 separate wars. These wars could communicate with each other via get a p=
ost requests. This sounds unconventional offhand so I'd like to hear some t=
houghts on it.
-Danny=20

> Date: Thu=2C 31 Oct 2013 13:28:41 -0700
> Subject: Re: [DISCUSS] Switching Streams from Camel deployment to .war de=
ployment
> From: chris@cxtsoftware.com
> To: dev@streams.incubator.apache.org
>=20
> Or Content Enricher [2]
>=20
> [2] http://camel.apache.org/content-enricher.html
>=20
>=20
> On Thu=2C Oct 31=2C 2013 at 12:37 PM=2C Jason Letourneau
> <jletourneau80@gmail.com>wrote:
>=20
> > check out the link here[1]
> >
> >
> > [1]http://camel.apache.org/request-reply.html
> >
> > On Thu=2C Oct 31=2C 2013 at 2:32 PM=2C Danny Sullivan <dsullivan7@hotma=
il.com>
> > wrote:
> > > I have a quick Camel question that I arrived at in the implementation=
 of
> > these new components:
> > >
> > > Lets say I have a method in streams-activity.jar that needs all
> > subscribers in the database. This would require a call to the
> > streams-persistence.jar. So far=2C I've seen camel used mostly for pass=
ing
> > data through the application=2C but not for making a single request-rep=
onse
> > from within a method. How can I use Camel to get a list of all subscrib=
ers
> > in the streams-persistence.jar from the streams-activity.jar?
> > >
> > >> From: dsullivan7@hotmail.com
> > >> To: dev@streams.incubator.apache.org
> > >> Subject: RE: [DISCUSS] Switching Streams from Camel deployment to .w=
ar
> > deployment
> > >> Date: Thu=2C 31 Oct 2013 08:40:44 -0400
> > >>
> > >> Excellent=2C I'll write up something as a proof of concept and we ca=
n
> > discuss further to make sure everything is vanilla.
> > >>
> > >> > Date: Wed=2C 30 Oct 2013 17:01:28 -0400
> > >> > Subject: Re: [DISCUSS] Switching Streams from Camel deployment to
> > .war deployment
> > >> > From: jletourneau80@gmail.com
> > >> > To: dev@streams.incubator.apache.org
> > >> >
> > >> > That sounds pretty promising to me.
> > >> >
> > >> > On Wed=2C Oct 30=2C 2013 at 2:38 PM=2C Danny Sullivan <
> > dsullivan7@hotmail.com> wrote:
> > >> > > Thanks for the feedback. You have an interesting point about the
> > url linking to a separate processing space. Let me tie my answer into y=
our
> > last question about "advocating for the simplicity at registration to g=
ive
> > up flexibility at registration=2C but retaining the inner "guts" of
> > EIP/messaging". Consider a new architecture:
> > >> > >
> > >> > > streams-web.war: single entry point to application=2C but functi=
ons
> > ONLY as an entry point. From here Camel routes the incoming requests to=
 4
> > separate jarssubscriber-registration.jar: subscriber registration
> > publisher-registration.jar: publisher registrationactivity.jar: returns
> > activity (also contains subscriber warehouse and storm activity
> > aggregator)publish.jar: publishes activitystreams-cassandra.jar: the ab=
ove
> > 4 jars would all have a hook into this jar which would function as a ho=
ok
> > onto the database. Each jar would have camel route output to this jar.
> > >> > >
> > >> > > In this implementation=2C Camel would no longer be the entry and=
 exit
> > point of a client to the application=2C but would handle the communicat=
ion
> > between components. The flow of activity through the application would =
be
> > method based in each jar. This would allow deployment on up to 6 differ=
ent
> > process spaces. However=2C this does not address that there is a single
> > server entry point=2C but I'm not sure if it was a concern in the first=
 place.
> > >> > >
> > >> > > My argument=2C at its basis=2C is that we should move away from =
using
> > Camel as the entry point to the application. I would be happy to mainta=
in
> > messaging between components.
> > >> > >
> > >> > >> Date: Wed=2C 30 Oct 2013 12:32:55 -0400
> > >> > >> Subject: Re: [DISCUSS] Switching Streams from Camel deployment =
to
> > .war deployment
> > >> > >> From: jletourneau80@gmail.com
> > >> > >> To: dev@streams.incubator.apache.org
> > >> > >>
> > >> > >> An interesting use case that I am holding onto is the ability f=
or
> > >> > >> publishers to register via a single URL (registration endpoint)=
=2C
> > but
> > >> > >> be sent a URL back to post to a different process space for act=
ual
> > >> > >> publishing.  The same is true on the subscriber front.  Current=
ly=2C
> > the
> > >> > >> Camel/EIP infrastructure abstracts this because different
> > components
> > >> > >> deployed in different process spaces handling the route creatio=
n
> > can
> > >> > >> just be bolted onto a running Streams instance without new
> > subs/pubs
> > >> > >> behaving any differently than existing.  This seems to be a
> > >> > >> potentially critical scaling point.  Is there a way to do this =
with
> > >> > >> the Spring solution?
> > >> > >>
> > >> > >> The persistence point is a good one=2C though I would classify =
that
> > as
> > >> > >> "not implemented" vs "not possible" (not that you were).
> > >> > >>
> > >> > >> I'm not married to Camel=2C I just like the EIP approach to bui=
lding
> > >> > >> something that is ultimately a messaging system.  There are kno=
wn
> > >> > >> patterns that solve at least a subset of the problems Streams i=
s
> > >> > >> trying to solve and implementations that can handle the load an=
d
> > I'll
> > >> > >> reiterate flexibility =3D=3D complexity almost always.
> > >> > >>
> > >> > >> It comes right back to the central question: Do you want
> > flexibility
> > >> > >> or simplicity?  It doesn't have to be black and white either I
> > don't
> > >> > >> think...
> > >> > >>
> > >> > >> More pointedly: Where should we give up flexibility for
> > simplicity?  I
> > >> > >> read that Danny is advocating for the simplicity at registratio=
n to
> > >> > >> give up flexibility at registration=2C but retaining the inner
> > "guts" of
> > >> > >> EIP/messaging?  Thoughts?
> > >> > >>
> > >> > >> On Wed=2C Oct 30=2C 2013 at 11:37 AM=2C Danny Sullivan <
> > dsullivan7@hotmail.com> wrote:
> > >> > >> > My argument is not for the IoC pattern as that can be (and ha=
s
> > been) implemented alongside Camel. My main argument is that the syntax =
at
> > the entry point is not only familiar but much simpler. This wouldn't be=
 a
> > very strong argument if the Camel implementation wasn't much more
> > complicated but I feel that it is the case. Also=2C looking toward the
> > future=2C if the server is restarted=2C in-routes are lost in Camel. Th=
e way to
> > curb this is to persist the dynamic routes that Camel creates=2C and th=
en on
> > start up pull every one of these routes and recreate a dynamic route fo=
r
> > each one. Not only is this much easier to implement using the Spring we=
b
> > implementation=2C but it already has been implemented and you can try i=
t by
> > checking out the webservice branch=2C registering a subscriber=2C resta=
rting
> > tomcat=2C and using the same url you had before. This will allow subscr=
ibers
> > to hang on to their urls once they register. (the same is true for
> > publishers: you can post via the same url after restarting tomcat)
> > >> > >> >
> > >> > >> >> Date: Wed=2C 30 Oct 2013 11:00:21 -0400
> > >> > >> >> Subject: Re: [DISCUSS] Switching Streams from Camel deployme=
nt
> > to .war deployment
> > >> > >> >> From: jletourneau80@gmail.com
> > >> > >> >> To: dev@streams.incubator.apache.org
> > >> > >> >>
> > >> > >> >> To be fair=2C while the current implementation is heavily
> > camel-based=2C
> > >> > >> >> all of the interfaces related to Streams functionality are n=
ot.
> >  The
> > >> > >> >> current model maps to what Matt has outlined in my opinion=
=2C
> > though
> > >> > >> >> packing names etc. probably don't follow that exact pattern.
> > >> > >> >>
> > >> > >> >> With regards to the complexity and different components in t=
he
> > >> > >> >> registration process=2C this was a cut at the abstraction ba=
sed
> > on the
> > >> > >> >> assumption that different implementations may be plugged in =
and
> > in
> > >> > >> >> fact may live on different processor space (ie. a polling
> > publisher vs
> > >> > >> >> a push publisher may be instantiated on different servers bu=
t
> > the
> > >> > >> >> registration URL is staticly defined).
> > >> > >> >>
> > >> > >> >> Is the main argument I am seeing for Spring  the familiarity=
 of
> > its
> > >> > >> >> IoC pattern implementation and syntax at the entry point?
> > >> > >> >>
> > >> > >> >> On Wed=2C Oct 30=2C 2013 at 10:53 AM=2C Danny Sullivan <
> > dsullivan7@hotmail.com> wrote:
> > >> > >> >> > Could you clarify whether the same entry points would exis=
t
> > for the camel implementation of the core (implementing the "process"
> > method/ using a DynammicRouteBuilder) or would the webservice be the so=
le
> > entry point to Streams and after it enters would it hand it off to Came=
l?
> > And what would be the entry point for the Storm implementation?
> > >> > >> >> > -Danny
> > >> > >> >> >
> > >> > >> >> >> Date: Wed=2C 30 Oct 2013 10:13:04 -0400
> > >> > >> >> >> Subject: Re: [DISCUSS] Switching Streams from Camel
> > deployment to .war deployment
> > >> > >> >> >> From: m.ben.franklin@gmail.com
> > >> > >> >> >> To: dev@streams.incubator.apache.org
> > >> > >> >> >>
> > >> > >> >> >> On Tue=2C Oct 29=2C 2013 at 2:55 PM=2C Danny Sullivan <
> > dsullivan7@hotmail.com>wrote:
> > >> > >> >> >>
> > >> > >> >> >> > My case for switching from OSGi is for simplicity in
> > design. To follow the
> > >> > >> >> >> > path of an activity through Streams in the webservice=
=2C
> > there is one main
> > >> > >> >> >> > things the developer needs to understand:
> > >> > >> >> >> > The @RequestMapping annotation specifies the HTTP entry
> > point
> > >> > >> >> >> > There are 4 @RequestMapping annotations that correspond=
 to
> > each of the 4
> > >> > >> >> >> > ways a user enters the application: registering a
> > publisher=2C registering a
> > >> > >> >> >> > subscriber=2C publishing activity=2C and getting an act=
ivity
> > stream. Where
> > >> > >> >> >> > these are located in the source code can be found by
> > searching for the
> > >> > >> >> >> > paths specified in the documentation (search for
> > "/publisherRegister"=2C
> > >> > >> >> >> > "/subscriberRegister"=2C "/publishActivity"=2C "/getAct=
ivity"
> > which will all
> > >> > >> >> >> > lead you to StreamsWebController.java). From the method=
s
> > that process
> > >> > >> >> >> > requests=2C the flow through the application is through
> > methods which can be
> > >> > >> >> >> > understood by most Java programmers.
> > >> > >> >> >> > The flow of activities through the current trunk branch=
 is
> > understood as
> > >> > >> >> >> > follows:
> > >> > >> >> >> > The string "/publisher/register" (the entry point to
> > register a publisher
> > >> > >> >> >> > specified in the documentation) is the value of the
> > >> > >> >> >> > consumer.registrationEndpoint property defined in
> > streams.propertiesThe
> > >> > >> >> >> > camelContext.xml specifies an endpoint with the id
> > >> > >> >> >> > consumerRegistrationEndpoint to have a uri equal to the
> > propertyThe
> > >> > >> >> >> > consumerRegistrationEndpoint routes from uri
> > direct:publisher register with
> > >> > >> >> >> > the bean activityRegistrationProcessor nested in betwee=
n
> > the routeThe
> > >> > >> >> >> > streams-eip-applicationContext contains a bean with the=
 id
> > >> > >> >> >> > activityRegistrationProcessor created for the class
> > >> > >> >> >> > ActivityPublisherRegistrationProcessorThe exchange will
> > enter the "process"
> > >> > >> >> >> > method of the class
> > ActivityPublisherRegisitrationProcessor and that this
> > >> > >> >> >> > is because this class implements the "Processor" interf=
ace
> > provided by
> > >> > >> >> >> > CamelThe direct:add-publisher-route takes the exchange
> > output from the
> > >> > >> >> >> > "process" method and routes it to the
> > activityRegistrationProcessor
> > >> > >> >> >> > "register" method. The bean activityRegistrationProcess=
or
> > is defined in the
> > >> > >> >> >> > streams-eip-osgi-component-import.xmlThe output from th=
is
> > method is then
> > >> > >> >> >> > sent to the "createNewRouteForConsumer" method of
> > activityConsumerRouter.
> > >> > >> >> >> > This method creates a new route for the newly registere=
d
> > publisher using
> > >> > >> >> >> > the private static final class DynamicConsumerRouteBuil=
der
> > which is
> > >> > >> >> >> > required to extend RouteBuilder which is provided by
> > Camel. This
> > >> > >> >> >> > DynamicConsumerRouteBuilder contains several methods:
> > >> > >> >> >> > getConsumerReceiveMethod() (which corresponds to @Value
> > >> > >> >> >> > ${consumer.receiveMethod} which corresponds to "receive=
")=2C
> > >> > >> >> >> > getConsumerSplitMethod() (@Value
> > ${consumer.splitMethod}=2C"split")=2C and
> > >> > >> >> >> > getConsumerActivityQUri() (@Value ${consumerActivityQUr=
i}=2C
> > >> > >> >> >> > "direct:activityQ"). This is different than the
> > camelContext.xml in that
> > >> > >> >> >> > the route is being created programmatically. What this =
is
> > doing is routing
> > >> > >> >> >> > input from the inroute url (which Camel does automatica=
lly
> > through the
> > >> > >> >> >> > configure method which is required to be overridden)=2C=
 to
> > the "receive"
> > >> > >> >> >> > method of ActivityConsumer=2C then to the "split" metho=
d of
> > ActivityConsumer=2C
> > >> > >> >> >> > and then to the "direct:activityQ" which if you look ba=
ck
> > in the
> > >> > >> >> >> > camelContext.xml routes to the "activemq:queue:activiti=
es"
> > which then
> > >> > >> >> >> > routes to "receiveExchange"
> > >> > >> >> >> > This is the process to register a publisher. The proces=
s
> > for registering a
> > >> > >> >> >> > subscriber is relatively the same though it involves
> > separate classes with
> > >> > >> >> >> > their own private static final RouteBuilder class. From=
 my
> > perspective=2C the
> > >> > >> >> >> > two most difficult things with setting this project up
> > were understanding
> > >> > >> >> >> > that the "process" method of the class that implements
> > "Processor" is the
> > >> > >> >> >> > entry point and the DynamicConsumerRouteBuilder creates
> > the second entry
> > >> > >> >> >> > point (The 5th and last points). This made the project
> > very=2C VERY hard to
> > >> > >> >> >> > understand.
> > >> > >> >> >> > In addition to simplicity of design=2C the mvn clean in=
stall
> > of the web
> > >> > >> >> >> > service project is much faster and small scale activity
> > publishing is also
> > >> > >> >> >> > faster (see my email about load testing). These are min=
or
> > points though as
> > >> > >> >> >> > compilation has no effect on deployment. OSGi does add =
the
> > benefit of
> > >> > >> >> >> > modularized programming which is valuable=2C though I t=
hink
> > the added
> > >> > >> >> >> > complexity of Camel merits moving the project away from
> > this paradigm.
> > >> > >> >> >> >
> > >> > >> >> >>
> > >> > >> >> >> I agree that the project is pretty difficult to understan=
d
> > ATM.  I think
> > >> > >> >> >> what we need to do is think about what the responsibiliti=
es
> > of the code are
> > >> > >> >> >> and allow for different implementations that are not so
> > tightly coupled as
> > >> > >> >> >> they are now.  For instance=2C having worked with Storm t=
o
> > ingest millions of
> > >> > >> >> >> activities a day=2C I personally would like to see stream=
s be
> > responsible for
> > >> > >> >> >> defining an over-arching orchestration model that can be
> > implemented within
> > >> > >> >> >> a single war or on top of a distributed system.  This wou=
ld
> > look something
> > >> > >> >> >> like the follows:
> > >> > >> >> >>
> > >> > >> >> >> |__ Streams-Core  (Base classes=2C interfaces=2C utilitie=
s=2C
> > extensions=2C etc)
> > >> > >> >> >> |
> > >> > >> >> >> |__ Streams-Storm (Storm implementation of the core)
> > >> > >> >> >> |
> > >> > >> >> >> |__ Streams-Camel (Camel implementation of the core)
> > >> > >> >> >> |
> > >> > >> >> >> |__ Streams-WS (Web service implementation)
> > >> > >> >> >>
> > >> > >> >> >>
> > >> > >> >> >> > Danny
> > >> > >> >> >> >
> > >> > >> >> >> >
> > >> > >> >> >> > > From: dsullivan7@hotmail.com
> > >> > >> >> >> > > To: dev@streams.incubator.apache.org
> > >> > >> >> >> > > Subject: [DISCUSS] Switching Streams from Camel
> > deployment to .war
> > >> > >> >> >> > deployment
> > >> > >> >> >> > > Date: Tue=2C 29 Oct 2013 11:07:39 -0400
> > >> > >> >> >> > >
> > >> > >> >> >> > > The discussion thread for switching Streams from
> > Camel/osgi/Servicemix
> > >> > >> >> >> > to a single .war deployment
> > >> > >> >> >> >
> > >> > >> >> >> >
> > >> > >> >> >
> > >> > >> >
> > >> > >
> > >>
> > >
> >
 		 	   		  =

--_9cb8a643-dbd8-4c37-be26-c644434cb86f_--