streams-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Danny Sullivan <dsulliv...@hotmail.com>
Subject RE: [DISCUSS] Switching Streams from Camel deployment to .war deployment
Date Thu, 31 Oct 2013 21:04:09 GMT
I'm not quite following this, so I apologize. What I'm trying to do is programmatically make
a request to a jar running on a separate jvm and get the response from that call all within
the same method. Similar to this http request:

HttpGet httpget = new HttpGet();

httpget.setURI(new URI("www.streams-persistence.com"));

CloseableHttpResponse response = httpclient.execute(httpget);

//do stuff with the response...




I imagine that this would translate to something in Camel similar to this:


<route>

<from uri="bean:subscriberService?method=getAllSubscribers"/>

<inOut uri="addressToStreamsPersistenceJar?method=selectAllSubscribersFromDatabase"/>

</route>


But it is unclear what the actual implementation would be.


This actually brings me to another suggestion. Would there be a big performance impact to
have communication between the software components occur between http? Say the 5 software
components I outlined earlier were packaged as 5 separate wars. These wars could communicate
with each other via get a post requests. This sounds unconventional offhand so I'd like to
hear some thoughts on it.
-Danny 

> Date: Thu, 31 Oct 2013 13:28:41 -0700
> Subject: Re: [DISCUSS] Switching Streams from Camel deployment to .war deployment
> From: chris@cxtsoftware.com
> To: dev@streams.incubator.apache.org
> 
> Or Content Enricher [2]
> 
> [2] http://camel.apache.org/content-enricher.html
> 
> 
> On Thu, Oct 31, 2013 at 12:37 PM, Jason Letourneau
> <jletourneau80@gmail.com>wrote:
> 
> > check out the link here[1]
> >
> >
> > [1]http://camel.apache.org/request-reply.html
> >
> > On Thu, Oct 31, 2013 at 2:32 PM, Danny Sullivan <dsullivan7@hotmail.com>
> > wrote:
> > > I have a quick Camel question that I arrived at in the implementation of
> > these new components:
> > >
> > > Lets say I have a method in streams-activity.jar that needs all
> > subscribers in the database. This would require a call to the
> > streams-persistence.jar. So far, I've seen camel used mostly for passing
> > data through the application, but not for making a single request-reponse
> > from within a method. How can I use Camel to get a list of all subscribers
> > in the streams-persistence.jar from the streams-activity.jar?
> > >
> > >> From: dsullivan7@hotmail.com
> > >> To: dev@streams.incubator.apache.org
> > >> Subject: RE: [DISCUSS] Switching Streams from Camel deployment to .war
> > deployment
> > >> Date: Thu, 31 Oct 2013 08:40:44 -0400
> > >>
> > >> Excellent, I'll write up something as a proof of concept and we can
> > discuss further to make sure everything is vanilla.
> > >>
> > >> > Date: Wed, 30 Oct 2013 17:01:28 -0400
> > >> > Subject: Re: [DISCUSS] Switching Streams from Camel deployment to
> > .war deployment
> > >> > From: jletourneau80@gmail.com
> > >> > To: dev@streams.incubator.apache.org
> > >> >
> > >> > That sounds pretty promising to me.
> > >> >
> > >> > On Wed, Oct 30, 2013 at 2:38 PM, Danny Sullivan <
> > dsullivan7@hotmail.com> wrote:
> > >> > > Thanks for the feedback. You have an interesting point about
the
> > url linking to a separate processing space. Let me tie my answer into your
> > last question about "advocating for the simplicity at registration to give
> > up flexibility at registration, but retaining the inner "guts" of
> > EIP/messaging". Consider a new architecture:
> > >> > >
> > >> > > streams-web.war: single entry point to application, but functions
> > ONLY as an entry point. From here Camel routes the incoming requests to 4
> > separate jarssubscriber-registration.jar: subscriber registration
> > publisher-registration.jar: publisher registrationactivity.jar: returns
> > activity (also contains subscriber warehouse and storm activity
> > aggregator)publish.jar: publishes activitystreams-cassandra.jar: the above
> > 4 jars would all have a hook into this jar which would function as a hook
> > onto the database. Each jar would have camel route output to this jar.
> > >> > >
> > >> > > In this implementation, Camel would no longer be the entry and
exit
> > point of a client to the application, but would handle the communication
> > between components. The flow of activity through the application would be
> > method based in each jar. This would allow deployment on up to 6 different
> > process spaces. However, this does not address that there is a single
> > server entry point, but I'm not sure if it was a concern in the first place.
> > >> > >
> > >> > > My argument, at its basis, is that we should move away from using
> > Camel as the entry point to the application. I would be happy to maintain
> > messaging between components.
> > >> > >
> > >> > >> Date: Wed, 30 Oct 2013 12:32:55 -0400
> > >> > >> Subject: Re: [DISCUSS] Switching Streams from Camel deployment
to
> > .war deployment
> > >> > >> From: jletourneau80@gmail.com
> > >> > >> To: dev@streams.incubator.apache.org
> > >> > >>
> > >> > >> An interesting use case that I am holding onto is the ability
for
> > >> > >> publishers to register via a single URL (registration endpoint),
> > but
> > >> > >> be sent a URL back to post to a different process space for
actual
> > >> > >> publishing.  The same is true on the subscriber front.  Currently,
> > the
> > >> > >> Camel/EIP infrastructure abstracts this because different
> > components
> > >> > >> deployed in different process spaces handling the route creation
> > can
> > >> > >> just be bolted onto a running Streams instance without new
> > subs/pubs
> > >> > >> behaving any differently than existing.  This seems to be
a
> > >> > >> potentially critical scaling point.  Is there a way to do
this with
> > >> > >> the Spring solution?
> > >> > >>
> > >> > >> The persistence point is a good one, though I would classify
that
> > as
> > >> > >> "not implemented" vs "not possible" (not that you were).
> > >> > >>
> > >> > >> I'm not married to Camel, I just like the EIP approach to
building
> > >> > >> something that is ultimately a messaging system.  There are
known
> > >> > >> patterns that solve at least a subset of the problems Streams
is
> > >> > >> trying to solve and implementations that can handle the load
and
> > I'll
> > >> > >> reiterate flexibility == complexity almost always.
> > >> > >>
> > >> > >> It comes right back to the central question: Do you want
> > flexibility
> > >> > >> or simplicity?  It doesn't have to be black and white either
I
> > don't
> > >> > >> think...
> > >> > >>
> > >> > >> More pointedly: Where should we give up flexibility for
> > simplicity?  I
> > >> > >> read that Danny is advocating for the simplicity at registration
to
> > >> > >> give up flexibility at registration, but retaining the inner
> > "guts" of
> > >> > >> EIP/messaging?  Thoughts?
> > >> > >>
> > >> > >> On Wed, Oct 30, 2013 at 11:37 AM, Danny Sullivan <
> > dsullivan7@hotmail.com> wrote:
> > >> > >> > My argument is not for the IoC pattern as that can be
(and has
> > been) implemented alongside Camel. My main argument is that the syntax at
> > the entry point is not only familiar but much simpler. This wouldn't be a
> > very strong argument if the Camel implementation wasn't much more
> > complicated but I feel that it is the case. Also, looking toward the
> > future, if the server is restarted, in-routes are lost in Camel. The way to
> > curb this is to persist the dynamic routes that Camel creates, and then on
> > start up pull every one of these routes and recreate a dynamic route for
> > each one. Not only is this much easier to implement using the Spring web
> > implementation, but it already has been implemented and you can try it by
> > checking out the webservice branch, registering a subscriber, restarting
> > tomcat, and using the same url you had before. This will allow subscribers
> > to hang on to their urls once they register. (the same is true for
> > publishers: you can post via the same url after restarting tomcat)
> > >> > >> >
> > >> > >> >> Date: Wed, 30 Oct 2013 11:00:21 -0400
> > >> > >> >> Subject: Re: [DISCUSS] Switching Streams from Camel
deployment
> > to .war deployment
> > >> > >> >> From: jletourneau80@gmail.com
> > >> > >> >> To: dev@streams.incubator.apache.org
> > >> > >> >>
> > >> > >> >> To be fair, while the current implementation is
heavily
> > camel-based,
> > >> > >> >> all of the interfaces related to Streams functionality
are not.
> >  The
> > >> > >> >> current model maps to what Matt has outlined in
my opinion,
> > though
> > >> > >> >> packing names etc. probably don't follow that exact
pattern.
> > >> > >> >>
> > >> > >> >> With regards to the complexity and different components
in the
> > >> > >> >> registration process, this was a cut at the abstraction
based
> > on the
> > >> > >> >> assumption that different implementations may be
plugged in and
> > in
> > >> > >> >> fact may live on different processor space (ie.
a polling
> > publisher vs
> > >> > >> >> a push publisher may be instantiated on different
servers but
> > the
> > >> > >> >> registration URL is staticly defined).
> > >> > >> >>
> > >> > >> >> Is the main argument I am seeing for Spring  the
familiarity of
> > its
> > >> > >> >> IoC pattern implementation and syntax at the entry
point?
> > >> > >> >>
> > >> > >> >> On Wed, Oct 30, 2013 at 10:53 AM, Danny Sullivan
<
> > dsullivan7@hotmail.com> wrote:
> > >> > >> >> > Could you clarify whether the same entry points
would exist
> > for the camel implementation of the core (implementing the "process"
> > method/ using a DynammicRouteBuilder) or would the webservice be the sole
> > entry point to Streams and after it enters would it hand it off to Camel?
> > And what would be the entry point for the Storm implementation?
> > >> > >> >> > -Danny
> > >> > >> >> >
> > >> > >> >> >> Date: Wed, 30 Oct 2013 10:13:04 -0400
> > >> > >> >> >> Subject: Re: [DISCUSS] Switching Streams
from Camel
> > deployment to .war deployment
> > >> > >> >> >> From: m.ben.franklin@gmail.com
> > >> > >> >> >> To: dev@streams.incubator.apache.org
> > >> > >> >> >>
> > >> > >> >> >> On Tue, Oct 29, 2013 at 2:55 PM, Danny
Sullivan <
> > dsullivan7@hotmail.com>wrote:
> > >> > >> >> >>
> > >> > >> >> >> > My case for switching from OSGi is
for simplicity in
> > design. To follow the
> > >> > >> >> >> > path of an activity through Streams
in the webservice,
> > there is one main
> > >> > >> >> >> > things the developer needs to understand:
> > >> > >> >> >> > The @RequestMapping annotation specifies
the HTTP entry
> > point
> > >> > >> >> >> > There are 4 @RequestMapping annotations
that correspond to
> > each of the 4
> > >> > >> >> >> > ways a user enters the application:
registering a
> > publisher, registering a
> > >> > >> >> >> > subscriber, publishing activity, and
getting an activity
> > stream. Where
> > >> > >> >> >> > these are located in the source code
can be found by
> > searching for the
> > >> > >> >> >> > paths specified in the documentation
(search for
> > "/publisherRegister",
> > >> > >> >> >> > "/subscriberRegister", "/publishActivity",
"/getActivity"
> > which will all
> > >> > >> >> >> > lead you to StreamsWebController.java).
From the methods
> > that process
> > >> > >> >> >> > requests, the flow through the application
is through
> > methods which can be
> > >> > >> >> >> > understood by most Java programmers.
> > >> > >> >> >> > The flow of activities through the
current trunk branch is
> > understood as
> > >> > >> >> >> > follows:
> > >> > >> >> >> > The string "/publisher/register" (the
entry point to
> > register a publisher
> > >> > >> >> >> > specified in the documentation) is
the value of the
> > >> > >> >> >> > consumer.registrationEndpoint property
defined in
> > streams.propertiesThe
> > >> > >> >> >> > camelContext.xml specifies an endpoint
with the id
> > >> > >> >> >> > consumerRegistrationEndpoint to have
a uri equal to the
> > propertyThe
> > >> > >> >> >> > consumerRegistrationEndpoint routes
from uri
> > direct:publisher register with
> > >> > >> >> >> > the bean activityRegistrationProcessor
nested in between
> > the routeThe
> > >> > >> >> >> > streams-eip-applicationContext contains
a bean with the id
> > >> > >> >> >> > activityRegistrationProcessor created
for the class
> > >> > >> >> >> > ActivityPublisherRegistrationProcessorThe
exchange will
> > enter the "process"
> > >> > >> >> >> > method of the class
> > ActivityPublisherRegisitrationProcessor and that this
> > >> > >> >> >> > is because this class implements the
"Processor" interface
> > provided by
> > >> > >> >> >> > CamelThe direct:add-publisher-route
takes the exchange
> > output from the
> > >> > >> >> >> > "process" method and routes it to
the
> > activityRegistrationProcessor
> > >> > >> >> >> > "register" method. The bean activityRegistrationProcessor
> > is defined in the
> > >> > >> >> >> > streams-eip-osgi-component-import.xmlThe
output from this
> > method is then
> > >> > >> >> >> > sent to the "createNewRouteForConsumer"
method of
> > activityConsumerRouter.
> > >> > >> >> >> > This method creates a new route for
the newly registered
> > publisher using
> > >> > >> >> >> > the private static final class DynamicConsumerRouteBuilder
> > which is
> > >> > >> >> >> > required to extend RouteBuilder which
is provided by
> > Camel. This
> > >> > >> >> >> > DynamicConsumerRouteBuilder contains
several methods:
> > >> > >> >> >> > getConsumerReceiveMethod() (which
corresponds to @Value
> > >> > >> >> >> > ${consumer.receiveMethod} which corresponds
to "receive"),
> > >> > >> >> >> > getConsumerSplitMethod() (@Value
> > ${consumer.splitMethod},"split"), and
> > >> > >> >> >> > getConsumerActivityQUri() (@Value
${consumerActivityQUri},
> > >> > >> >> >> > "direct:activityQ"). This is different
than the
> > camelContext.xml in that
> > >> > >> >> >> > the route is being created programmatically.
What this is
> > doing is routing
> > >> > >> >> >> > input from the inroute url (which
Camel does automatically
> > through the
> > >> > >> >> >> > configure method which is required
to be overridden), to
> > the "receive"
> > >> > >> >> >> > method of ActivityConsumer, then to
the "split" method of
> > ActivityConsumer,
> > >> > >> >> >> > and then to the "direct:activityQ"
which if you look back
> > in the
> > >> > >> >> >> > camelContext.xml routes to the "activemq:queue:activities"
> > which then
> > >> > >> >> >> > routes to "receiveExchange"
> > >> > >> >> >> > This is the process to register a
publisher. The process
> > for registering a
> > >> > >> >> >> > subscriber is relatively the same
though it involves
> > separate classes with
> > >> > >> >> >> > their own private static final RouteBuilder
class. From my
> > perspective, the
> > >> > >> >> >> > two most difficult things with setting
this project up
> > were understanding
> > >> > >> >> >> > that the "process" method of the class
that implements
> > "Processor" is the
> > >> > >> >> >> > entry point and the DynamicConsumerRouteBuilder
creates
> > the second entry
> > >> > >> >> >> > point (The 5th and last points). This
made the project
> > very, VERY hard to
> > >> > >> >> >> > understand.
> > >> > >> >> >> > In addition to simplicity of design,
the mvn clean install
> > of the web
> > >> > >> >> >> > service project is much faster and
small scale activity
> > publishing is also
> > >> > >> >> >> > faster (see my email about load testing).
These are minor
> > points though as
> > >> > >> >> >> > compilation has no effect on deployment.
OSGi does add the
> > benefit of
> > >> > >> >> >> > modularized programming which is valuable,
though I think
> > the added
> > >> > >> >> >> > complexity of Camel merits moving
the project away from
> > this paradigm.
> > >> > >> >> >> >
> > >> > >> >> >>
> > >> > >> >> >> I agree that the project is pretty difficult
to understand
> > ATM.  I think
> > >> > >> >> >> what we need to do is think about what
the responsibilities
> > of the code are
> > >> > >> >> >> and allow for different implementations
that are not so
> > tightly coupled as
> > >> > >> >> >> they are now.  For instance, having worked
with Storm to
> > ingest millions of
> > >> > >> >> >> activities a day, I personally would like
to see streams be
> > responsible for
> > >> > >> >> >> defining an over-arching orchestration
model that can be
> > implemented within
> > >> > >> >> >> a single war or on top of a distributed
system.  This would
> > look something
> > >> > >> >> >> like the follows:
> > >> > >> >> >>
> > >> > >> >> >> |__ Streams-Core  (Base classes, interfaces,
utilities,
> > extensions, etc)
> > >> > >> >> >> |
> > >> > >> >> >> |__ Streams-Storm (Storm implementation
of the core)
> > >> > >> >> >> |
> > >> > >> >> >> |__ Streams-Camel (Camel implementation
of the core)
> > >> > >> >> >> |
> > >> > >> >> >> |__ Streams-WS (Web service implementation)
> > >> > >> >> >>
> > >> > >> >> >>
> > >> > >> >> >> > Danny
> > >> > >> >> >> >
> > >> > >> >> >> >
> > >> > >> >> >> > > From: dsullivan7@hotmail.com
> > >> > >> >> >> > > To: dev@streams.incubator.apache.org
> > >> > >> >> >> > > Subject: [DISCUSS] Switching
Streams from Camel
> > deployment to .war
> > >> > >> >> >> > deployment
> > >> > >> >> >> > > Date: Tue, 29 Oct 2013 11:07:39
-0400
> > >> > >> >> >> > >
> > >> > >> >> >> > > The discussion thread for switching
Streams from
> > Camel/osgi/Servicemix
> > >> > >> >> >> > to a single .war deployment
> > >> > >> >> >> >
> > >> > >> >> >> >
> > >> > >> >> >
> > >> > >> >
> > >> > >
> > >>
> > >
> >
 		 	   		  
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message