streams-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason Letourneau <jletournea...@gmail.com>
Subject Re: [DISCUSS] Switching Streams from Camel deployment to .war deployment
Date Thu, 31 Oct 2013 19:37:17 GMT
check out the link here[1]


[1]http://camel.apache.org/request-reply.html

On Thu, Oct 31, 2013 at 2:32 PM, Danny Sullivan <dsullivan7@hotmail.com> wrote:
> I have a quick Camel question that I arrived at in the implementation of these new components:
>
> Lets say I have a method in streams-activity.jar that needs all subscribers in the database.
This would require a call to the streams-persistence.jar. So far, I've seen camel used mostly
for passing data through the application, but not for making a single request-reponse from
within a method. How can I use Camel to get a list of all subscribers in the streams-persistence.jar
from the streams-activity.jar?
>
>> From: dsullivan7@hotmail.com
>> To: dev@streams.incubator.apache.org
>> Subject: RE: [DISCUSS] Switching Streams from Camel deployment to .war deployment
>> Date: Thu, 31 Oct 2013 08:40:44 -0400
>>
>> Excellent, I'll write up something as a proof of concept and we can discuss further
to make sure everything is vanilla.
>>
>> > Date: Wed, 30 Oct 2013 17:01:28 -0400
>> > Subject: Re: [DISCUSS] Switching Streams from Camel deployment to .war deployment
>> > From: jletourneau80@gmail.com
>> > To: dev@streams.incubator.apache.org
>> >
>> > That sounds pretty promising to me.
>> >
>> > On Wed, Oct 30, 2013 at 2:38 PM, Danny Sullivan <dsullivan7@hotmail.com>
wrote:
>> > > Thanks for the feedback. You have an interesting point about the url linking
to a separate processing space. Let me tie my answer into your last question about "advocating
for the simplicity at registration to give up flexibility at registration, but retaining the
inner "guts" of EIP/messaging". Consider a new architecture:
>> > >
>> > > streams-web.war: single entry point to application, but functions ONLY
as an entry point. From here Camel routes the incoming requests to 4 separate jarssubscriber-registration.jar:
subscriber registration publisher-registration.jar: publisher registrationactivity.jar: returns
activity (also contains subscriber warehouse and storm activity aggregator)publish.jar: publishes
activitystreams-cassandra.jar: the above 4 jars would all have a hook into this jar which
would function as a hook onto the database. Each jar would have camel route output to this
jar.
>> > >
>> > > In this implementation, Camel would no longer be the entry and exit point
of a client to the application, but would handle the communication between components. The
flow of activity through the application would be method based in each jar. This would allow
deployment on up to 6 different process spaces. However, this does not address that there
is a single server entry point, but I'm not sure if it was a concern in the first place.
>> > >
>> > > My argument, at its basis, is that we should move away from using Camel
as the entry point to the application. I would be happy to maintain messaging between components.
>> > >
>> > >> Date: Wed, 30 Oct 2013 12:32:55 -0400
>> > >> Subject: Re: [DISCUSS] Switching Streams from Camel deployment to .war
deployment
>> > >> From: jletourneau80@gmail.com
>> > >> To: dev@streams.incubator.apache.org
>> > >>
>> > >> An interesting use case that I am holding onto is the ability for
>> > >> publishers to register via a single URL (registration endpoint), but
>> > >> be sent a URL back to post to a different process space for actual
>> > >> publishing.  The same is true on the subscriber front.  Currently,
the
>> > >> Camel/EIP infrastructure abstracts this because different components
>> > >> deployed in different process spaces handling the route creation can
>> > >> just be bolted onto a running Streams instance without new subs/pubs
>> > >> behaving any differently than existing.  This seems to be a
>> > >> potentially critical scaling point.  Is there a way to do this with
>> > >> the Spring solution?
>> > >>
>> > >> The persistence point is a good one, though I would classify that as
>> > >> "not implemented" vs "not possible" (not that you were).
>> > >>
>> > >> I'm not married to Camel, I just like the EIP approach to building
>> > >> something that is ultimately a messaging system.  There are known
>> > >> patterns that solve at least a subset of the problems Streams is
>> > >> trying to solve and implementations that can handle the load and I'll
>> > >> reiterate flexibility == complexity almost always.
>> > >>
>> > >> It comes right back to the central question: Do you want flexibility
>> > >> or simplicity?  It doesn't have to be black and white either I don't
>> > >> think...
>> > >>
>> > >> More pointedly: Where should we give up flexibility for simplicity?
 I
>> > >> read that Danny is advocating for the simplicity at registration to
>> > >> give up flexibility at registration, but retaining the inner "guts"
of
>> > >> EIP/messaging?  Thoughts?
>> > >>
>> > >> On Wed, Oct 30, 2013 at 11:37 AM, Danny Sullivan <dsullivan7@hotmail.com>
wrote:
>> > >> > My argument is not for the IoC pattern as that can be (and has
been) implemented alongside Camel. My main argument is that the syntax at the entry point
is not only familiar but much simpler. This wouldn't be a very strong argument if the Camel
implementation wasn't much more complicated but I feel that it is the case. Also, looking
toward the future, if the server is restarted, in-routes are lost in Camel. The way to curb
this is to persist the dynamic routes that Camel creates, and then on start up pull every
one of these routes and recreate a dynamic route for each one. Not only is this much easier
to implement using the Spring web implementation, but it already has been implemented and
you can try it by checking out the webservice branch, registering a subscriber, restarting
tomcat, and using the same url you had before. This will allow subscribers to hang on to their
urls once they register. (the same is true for publishers: you can post via the same url after
restarting tomcat)
>> > >> >
>> > >> >> Date: Wed, 30 Oct 2013 11:00:21 -0400
>> > >> >> Subject: Re: [DISCUSS] Switching Streams from Camel deployment
to .war deployment
>> > >> >> From: jletourneau80@gmail.com
>> > >> >> To: dev@streams.incubator.apache.org
>> > >> >>
>> > >> >> To be fair, while the current implementation is heavily camel-based,
>> > >> >> all of the interfaces related to Streams functionality are
not.  The
>> > >> >> current model maps to what Matt has outlined in my opinion,
though
>> > >> >> packing names etc. probably don't follow that exact pattern.
>> > >> >>
>> > >> >> With regards to the complexity and different components in
the
>> > >> >> registration process, this was a cut at the abstraction based
on the
>> > >> >> assumption that different implementations may be plugged in
and in
>> > >> >> fact may live on different processor space (ie. a polling
publisher vs
>> > >> >> a push publisher may be instantiated on different servers
but the
>> > >> >> registration URL is staticly defined).
>> > >> >>
>> > >> >> Is the main argument I am seeing for Spring  the familiarity
of its
>> > >> >> IoC pattern implementation and syntax at the entry point?
>> > >> >>
>> > >> >> On Wed, Oct 30, 2013 at 10:53 AM, Danny Sullivan <dsullivan7@hotmail.com>
wrote:
>> > >> >> > Could you clarify whether the same entry points would
exist for the camel implementation of the core (implementing the "process" method/ using a
DynammicRouteBuilder) or would the webservice be the sole entry point to Streams and after
it enters would it hand it off to Camel? And what would be the entry point for the Storm implementation?
>> > >> >> > -Danny
>> > >> >> >
>> > >> >> >> Date: Wed, 30 Oct 2013 10:13:04 -0400
>> > >> >> >> Subject: Re: [DISCUSS] Switching Streams from Camel
deployment to .war deployment
>> > >> >> >> From: m.ben.franklin@gmail.com
>> > >> >> >> To: dev@streams.incubator.apache.org
>> > >> >> >>
>> > >> >> >> On Tue, Oct 29, 2013 at 2:55 PM, Danny Sullivan <dsullivan7@hotmail.com>wrote:
>> > >> >> >>
>> > >> >> >> > My case for switching from OSGi is for simplicity
in design. To follow the
>> > >> >> >> > path of an activity through Streams in the webservice,
there is one main
>> > >> >> >> > things the developer needs to understand:
>> > >> >> >> > The @RequestMapping annotation specifies the
HTTP entry point
>> > >> >> >> > There are 4 @RequestMapping annotations that
correspond to each of the 4
>> > >> >> >> > ways a user enters the application: registering
a publisher, registering a
>> > >> >> >> > subscriber, publishing activity, and getting
an activity stream. Where
>> > >> >> >> > these are located in the source code can be
found by searching for the
>> > >> >> >> > paths specified in the documentation (search
for "/publisherRegister",
>> > >> >> >> > "/subscriberRegister", "/publishActivity", "/getActivity"
which will all
>> > >> >> >> > lead you to StreamsWebController.java). From
the methods that process
>> > >> >> >> > requests, the flow through the application is
through methods which can be
>> > >> >> >> > understood by most Java programmers.
>> > >> >> >> > The flow of activities through the current trunk
branch is understood as
>> > >> >> >> > follows:
>> > >> >> >> > The string "/publisher/register" (the entry
point to register a publisher
>> > >> >> >> > specified in the documentation) is the value
of the
>> > >> >> >> > consumer.registrationEndpoint property defined
in streams.propertiesThe
>> > >> >> >> > camelContext.xml specifies an endpoint with
the id
>> > >> >> >> > consumerRegistrationEndpoint to have a uri equal
to the propertyThe
>> > >> >> >> > consumerRegistrationEndpoint routes from uri
direct:publisher register with
>> > >> >> >> > the bean activityRegistrationProcessor nested
in between the routeThe
>> > >> >> >> > streams-eip-applicationContext contains a bean
with the id
>> > >> >> >> > activityRegistrationProcessor created for the
class
>> > >> >> >> > ActivityPublisherRegistrationProcessorThe exchange
will enter the "process"
>> > >> >> >> > method of the class ActivityPublisherRegisitrationProcessor
and that this
>> > >> >> >> > is because this class implements the "Processor"
interface provided by
>> > >> >> >> > CamelThe direct:add-publisher-route takes the
exchange output from the
>> > >> >> >> > "process" method and routes it to the activityRegistrationProcessor
>> > >> >> >> > "register" method. The bean activityRegistrationProcessor
is defined in the
>> > >> >> >> > streams-eip-osgi-component-import.xmlThe output
from this method is then
>> > >> >> >> > sent to the "createNewRouteForConsumer" method
of activityConsumerRouter.
>> > >> >> >> > This method creates a new route for the newly
registered publisher using
>> > >> >> >> > the private static final class DynamicConsumerRouteBuilder
which is
>> > >> >> >> > required to extend RouteBuilder which is provided
by Camel. This
>> > >> >> >> > DynamicConsumerRouteBuilder contains several
methods:
>> > >> >> >> > getConsumerReceiveMethod() (which corresponds
to @Value
>> > >> >> >> > ${consumer.receiveMethod} which corresponds
to "receive"),
>> > >> >> >> > getConsumerSplitMethod() (@Value ${consumer.splitMethod},"split"),
and
>> > >> >> >> > getConsumerActivityQUri() (@Value ${consumerActivityQUri},
>> > >> >> >> > "direct:activityQ"). This is different than
the camelContext.xml in that
>> > >> >> >> > the route is being created programmatically.
What this is doing is routing
>> > >> >> >> > input from the inroute url (which Camel does
automatically through the
>> > >> >> >> > configure method which is required to be overridden),
to the "receive"
>> > >> >> >> > method of ActivityConsumer, then to the "split"
method of ActivityConsumer,
>> > >> >> >> > and then to the "direct:activityQ" which if
you look back in the
>> > >> >> >> > camelContext.xml routes to the "activemq:queue:activities"
which then
>> > >> >> >> > routes to "receiveExchange"
>> > >> >> >> > This is the process to register a publisher.
The process for registering a
>> > >> >> >> > subscriber is relatively the same though it
involves separate classes with
>> > >> >> >> > their own private static final RouteBuilder
class. From my perspective, the
>> > >> >> >> > two most difficult things with setting this
project up were understanding
>> > >> >> >> > that the "process" method of the class that
implements "Processor" is the
>> > >> >> >> > entry point and the DynamicConsumerRouteBuilder
creates the second entry
>> > >> >> >> > point (The 5th and last points). This made the
project very, VERY hard to
>> > >> >> >> > understand.
>> > >> >> >> > In addition to simplicity of design, the mvn
clean install of the web
>> > >> >> >> > service project is much faster and small scale
activity publishing is also
>> > >> >> >> > faster (see my email about load testing). These
are minor points though as
>> > >> >> >> > compilation has no effect on deployment. OSGi
does add the benefit of
>> > >> >> >> > modularized programming which is valuable, though
I think the added
>> > >> >> >> > complexity of Camel merits moving the project
away from this paradigm.
>> > >> >> >> >
>> > >> >> >>
>> > >> >> >> I agree that the project is pretty difficult to understand
ATM.  I think
>> > >> >> >> what we need to do is think about what the responsibilities
of the code are
>> > >> >> >> and allow for different implementations that are
not so tightly coupled as
>> > >> >> >> they are now.  For instance, having worked with Storm
to ingest millions of
>> > >> >> >> activities a day, I personally would like to see
streams be responsible for
>> > >> >> >> defining an over-arching orchestration model that
can be implemented within
>> > >> >> >> a single war or on top of a distributed system. 
This would look something
>> > >> >> >> like the follows:
>> > >> >> >>
>> > >> >> >> |__ Streams-Core  (Base classes, interfaces, utilities,
extensions, etc)
>> > >> >> >> |
>> > >> >> >> |__ Streams-Storm (Storm implementation of the core)
>> > >> >> >> |
>> > >> >> >> |__ Streams-Camel (Camel implementation of the core)
>> > >> >> >> |
>> > >> >> >> |__ Streams-WS (Web service implementation)
>> > >> >> >>
>> > >> >> >>
>> > >> >> >> > Danny
>> > >> >> >> >
>> > >> >> >> >
>> > >> >> >> > > From: dsullivan7@hotmail.com
>> > >> >> >> > > To: dev@streams.incubator.apache.org
>> > >> >> >> > > Subject: [DISCUSS] Switching Streams from
Camel deployment to .war
>> > >> >> >> > deployment
>> > >> >> >> > > Date: Tue, 29 Oct 2013 11:07:39 -0400
>> > >> >> >> > >
>> > >> >> >> > > The discussion thread for switching Streams
from Camel/osgi/Servicemix
>> > >> >> >> > to a single .war deployment
>> > >> >> >> >
>> > >> >> >> >
>> > >> >> >
>> > >> >
>> > >
>>
>

Mime
View raw message