streams-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Geer <ch...@cxtsoftware.com>
Subject Re: [DISCUSS] Switching Streams from Camel deployment to .war deployment
Date Thu, 31 Oct 2013 20:28:41 GMT
Or Content Enricher [2]

[2] http://camel.apache.org/content-enricher.html


On Thu, Oct 31, 2013 at 12:37 PM, Jason Letourneau
<jletourneau80@gmail.com>wrote:

> check out the link here[1]
>
>
> [1]http://camel.apache.org/request-reply.html
>
> On Thu, Oct 31, 2013 at 2:32 PM, Danny Sullivan <dsullivan7@hotmail.com>
> wrote:
> > I have a quick Camel question that I arrived at in the implementation of
> these new components:
> >
> > Lets say I have a method in streams-activity.jar that needs all
> subscribers in the database. This would require a call to the
> streams-persistence.jar. So far, I've seen camel used mostly for passing
> data through the application, but not for making a single request-reponse
> from within a method. How can I use Camel to get a list of all subscribers
> in the streams-persistence.jar from the streams-activity.jar?
> >
> >> From: dsullivan7@hotmail.com
> >> To: dev@streams.incubator.apache.org
> >> Subject: RE: [DISCUSS] Switching Streams from Camel deployment to .war
> deployment
> >> Date: Thu, 31 Oct 2013 08:40:44 -0400
> >>
> >> Excellent, I'll write up something as a proof of concept and we can
> discuss further to make sure everything is vanilla.
> >>
> >> > Date: Wed, 30 Oct 2013 17:01:28 -0400
> >> > Subject: Re: [DISCUSS] Switching Streams from Camel deployment to
> .war deployment
> >> > From: jletourneau80@gmail.com
> >> > To: dev@streams.incubator.apache.org
> >> >
> >> > That sounds pretty promising to me.
> >> >
> >> > On Wed, Oct 30, 2013 at 2:38 PM, Danny Sullivan <
> dsullivan7@hotmail.com> wrote:
> >> > > Thanks for the feedback. You have an interesting point about the
> url linking to a separate processing space. Let me tie my answer into your
> last question about "advocating for the simplicity at registration to give
> up flexibility at registration, but retaining the inner "guts" of
> EIP/messaging". Consider a new architecture:
> >> > >
> >> > > streams-web.war: single entry point to application, but functions
> ONLY as an entry point. From here Camel routes the incoming requests to 4
> separate jarssubscriber-registration.jar: subscriber registration
> publisher-registration.jar: publisher registrationactivity.jar: returns
> activity (also contains subscriber warehouse and storm activity
> aggregator)publish.jar: publishes activitystreams-cassandra.jar: the above
> 4 jars would all have a hook into this jar which would function as a hook
> onto the database. Each jar would have camel route output to this jar.
> >> > >
> >> > > In this implementation, Camel would no longer be the entry and exit
> point of a client to the application, but would handle the communication
> between components. The flow of activity through the application would be
> method based in each jar. This would allow deployment on up to 6 different
> process spaces. However, this does not address that there is a single
> server entry point, but I'm not sure if it was a concern in the first place.
> >> > >
> >> > > My argument, at its basis, is that we should move away from using
> Camel as the entry point to the application. I would be happy to maintain
> messaging between components.
> >> > >
> >> > >> Date: Wed, 30 Oct 2013 12:32:55 -0400
> >> > >> Subject: Re: [DISCUSS] Switching Streams from Camel deployment
to
> .war deployment
> >> > >> From: jletourneau80@gmail.com
> >> > >> To: dev@streams.incubator.apache.org
> >> > >>
> >> > >> An interesting use case that I am holding onto is the ability
for
> >> > >> publishers to register via a single URL (registration endpoint),
> but
> >> > >> be sent a URL back to post to a different process space for actual
> >> > >> publishing.  The same is true on the subscriber front.  Currently,
> the
> >> > >> Camel/EIP infrastructure abstracts this because different
> components
> >> > >> deployed in different process spaces handling the route creation
> can
> >> > >> just be bolted onto a running Streams instance without new
> subs/pubs
> >> > >> behaving any differently than existing.  This seems to be a
> >> > >> potentially critical scaling point.  Is there a way to do this
with
> >> > >> the Spring solution?
> >> > >>
> >> > >> The persistence point is a good one, though I would classify that
> as
> >> > >> "not implemented" vs "not possible" (not that you were).
> >> > >>
> >> > >> I'm not married to Camel, I just like the EIP approach to building
> >> > >> something that is ultimately a messaging system.  There are known
> >> > >> patterns that solve at least a subset of the problems Streams
is
> >> > >> trying to solve and implementations that can handle the load and
> I'll
> >> > >> reiterate flexibility == complexity almost always.
> >> > >>
> >> > >> It comes right back to the central question: Do you want
> flexibility
> >> > >> or simplicity?  It doesn't have to be black and white either I
> don't
> >> > >> think...
> >> > >>
> >> > >> More pointedly: Where should we give up flexibility for
> simplicity?  I
> >> > >> read that Danny is advocating for the simplicity at registration
to
> >> > >> give up flexibility at registration, but retaining the inner
> "guts" of
> >> > >> EIP/messaging?  Thoughts?
> >> > >>
> >> > >> On Wed, Oct 30, 2013 at 11:37 AM, Danny Sullivan <
> dsullivan7@hotmail.com> wrote:
> >> > >> > My argument is not for the IoC pattern as that can be (and
has
> been) implemented alongside Camel. My main argument is that the syntax at
> the entry point is not only familiar but much simpler. This wouldn't be a
> very strong argument if the Camel implementation wasn't much more
> complicated but I feel that it is the case. Also, looking toward the
> future, if the server is restarted, in-routes are lost in Camel. The way to
> curb this is to persist the dynamic routes that Camel creates, and then on
> start up pull every one of these routes and recreate a dynamic route for
> each one. Not only is this much easier to implement using the Spring web
> implementation, but it already has been implemented and you can try it by
> checking out the webservice branch, registering a subscriber, restarting
> tomcat, and using the same url you had before. This will allow subscribers
> to hang on to their urls once they register. (the same is true for
> publishers: you can post via the same url after restarting tomcat)
> >> > >> >
> >> > >> >> Date: Wed, 30 Oct 2013 11:00:21 -0400
> >> > >> >> Subject: Re: [DISCUSS] Switching Streams from Camel deployment
> to .war deployment
> >> > >> >> From: jletourneau80@gmail.com
> >> > >> >> To: dev@streams.incubator.apache.org
> >> > >> >>
> >> > >> >> To be fair, while the current implementation is heavily
> camel-based,
> >> > >> >> all of the interfaces related to Streams functionality
are not.
>  The
> >> > >> >> current model maps to what Matt has outlined in my opinion,
> though
> >> > >> >> packing names etc. probably don't follow that exact pattern.
> >> > >> >>
> >> > >> >> With regards to the complexity and different components
in the
> >> > >> >> registration process, this was a cut at the abstraction
based
> on the
> >> > >> >> assumption that different implementations may be plugged
in and
> in
> >> > >> >> fact may live on different processor space (ie. a polling
> publisher vs
> >> > >> >> a push publisher may be instantiated on different servers
but
> the
> >> > >> >> registration URL is staticly defined).
> >> > >> >>
> >> > >> >> Is the main argument I am seeing for Spring  the familiarity
of
> its
> >> > >> >> IoC pattern implementation and syntax at the entry point?
> >> > >> >>
> >> > >> >> On Wed, Oct 30, 2013 at 10:53 AM, Danny Sullivan <
> dsullivan7@hotmail.com> wrote:
> >> > >> >> > Could you clarify whether the same entry points
would exist
> for the camel implementation of the core (implementing the "process"
> method/ using a DynammicRouteBuilder) or would the webservice be the sole
> entry point to Streams and after it enters would it hand it off to Camel?
> And what would be the entry point for the Storm implementation?
> >> > >> >> > -Danny
> >> > >> >> >
> >> > >> >> >> Date: Wed, 30 Oct 2013 10:13:04 -0400
> >> > >> >> >> Subject: Re: [DISCUSS] Switching Streams from
Camel
> deployment to .war deployment
> >> > >> >> >> From: m.ben.franklin@gmail.com
> >> > >> >> >> To: dev@streams.incubator.apache.org
> >> > >> >> >>
> >> > >> >> >> On Tue, Oct 29, 2013 at 2:55 PM, Danny Sullivan
<
> dsullivan7@hotmail.com>wrote:
> >> > >> >> >>
> >> > >> >> >> > My case for switching from OSGi is for
simplicity in
> design. To follow the
> >> > >> >> >> > path of an activity through Streams in
the webservice,
> there is one main
> >> > >> >> >> > things the developer needs to understand:
> >> > >> >> >> > The @RequestMapping annotation specifies
the HTTP entry
> point
> >> > >> >> >> > There are 4 @RequestMapping annotations
that correspond to
> each of the 4
> >> > >> >> >> > ways a user enters the application: registering
a
> publisher, registering a
> >> > >> >> >> > subscriber, publishing activity, and getting
an activity
> stream. Where
> >> > >> >> >> > these are located in the source code can
be found by
> searching for the
> >> > >> >> >> > paths specified in the documentation (search
for
> "/publisherRegister",
> >> > >> >> >> > "/subscriberRegister", "/publishActivity",
"/getActivity"
> which will all
> >> > >> >> >> > lead you to StreamsWebController.java).
From the methods
> that process
> >> > >> >> >> > requests, the flow through the application
is through
> methods which can be
> >> > >> >> >> > understood by most Java programmers.
> >> > >> >> >> > The flow of activities through the current
trunk branch is
> understood as
> >> > >> >> >> > follows:
> >> > >> >> >> > The string "/publisher/register" (the entry
point to
> register a publisher
> >> > >> >> >> > specified in the documentation) is the
value of the
> >> > >> >> >> > consumer.registrationEndpoint property
defined in
> streams.propertiesThe
> >> > >> >> >> > camelContext.xml specifies an endpoint
with the id
> >> > >> >> >> > consumerRegistrationEndpoint to have a
uri equal to the
> propertyThe
> >> > >> >> >> > consumerRegistrationEndpoint routes from
uri
> direct:publisher register with
> >> > >> >> >> > the bean activityRegistrationProcessor
nested in between
> the routeThe
> >> > >> >> >> > streams-eip-applicationContext contains
a bean with the id
> >> > >> >> >> > activityRegistrationProcessor created for
the class
> >> > >> >> >> > ActivityPublisherRegistrationProcessorThe
exchange will
> enter the "process"
> >> > >> >> >> > method of the class
> ActivityPublisherRegisitrationProcessor and that this
> >> > >> >> >> > is because this class implements the "Processor"
interface
> provided by
> >> > >> >> >> > CamelThe direct:add-publisher-route takes
the exchange
> output from the
> >> > >> >> >> > "process" method and routes it to the
> activityRegistrationProcessor
> >> > >> >> >> > "register" method. The bean activityRegistrationProcessor
> is defined in the
> >> > >> >> >> > streams-eip-osgi-component-import.xmlThe
output from this
> method is then
> >> > >> >> >> > sent to the "createNewRouteForConsumer"
method of
> activityConsumerRouter.
> >> > >> >> >> > This method creates a new route for the
newly registered
> publisher using
> >> > >> >> >> > the private static final class DynamicConsumerRouteBuilder
> which is
> >> > >> >> >> > required to extend RouteBuilder which is
provided by
> Camel. This
> >> > >> >> >> > DynamicConsumerRouteBuilder contains several
methods:
> >> > >> >> >> > getConsumerReceiveMethod() (which corresponds
to @Value
> >> > >> >> >> > ${consumer.receiveMethod} which corresponds
to "receive"),
> >> > >> >> >> > getConsumerSplitMethod() (@Value
> ${consumer.splitMethod},"split"), and
> >> > >> >> >> > getConsumerActivityQUri() (@Value ${consumerActivityQUri},
> >> > >> >> >> > "direct:activityQ"). This is different
than the
> camelContext.xml in that
> >> > >> >> >> > the route is being created programmatically.
What this is
> doing is routing
> >> > >> >> >> > input from the inroute url (which Camel
does automatically
> through the
> >> > >> >> >> > configure method which is required to be
overridden), to
> the "receive"
> >> > >> >> >> > method of ActivityConsumer, then to the
"split" method of
> ActivityConsumer,
> >> > >> >> >> > and then to the "direct:activityQ" which
if you look back
> in the
> >> > >> >> >> > camelContext.xml routes to the "activemq:queue:activities"
> which then
> >> > >> >> >> > routes to "receiveExchange"
> >> > >> >> >> > This is the process to register a publisher.
The process
> for registering a
> >> > >> >> >> > subscriber is relatively the same though
it involves
> separate classes with
> >> > >> >> >> > their own private static final RouteBuilder
class. From my
> perspective, the
> >> > >> >> >> > two most difficult things with setting
this project up
> were understanding
> >> > >> >> >> > that the "process" method of the class
that implements
> "Processor" is the
> >> > >> >> >> > entry point and the DynamicConsumerRouteBuilder
creates
> the second entry
> >> > >> >> >> > point (The 5th and last points). This made
the project
> very, VERY hard to
> >> > >> >> >> > understand.
> >> > >> >> >> > In addition to simplicity of design, the
mvn clean install
> of the web
> >> > >> >> >> > service project is much faster and small
scale activity
> publishing is also
> >> > >> >> >> > faster (see my email about load testing).
These are minor
> points though as
> >> > >> >> >> > compilation has no effect on deployment.
OSGi does add the
> benefit of
> >> > >> >> >> > modularized programming which is valuable,
though I think
> the added
> >> > >> >> >> > complexity of Camel merits moving the project
away from
> this paradigm.
> >> > >> >> >> >
> >> > >> >> >>
> >> > >> >> >> I agree that the project is pretty difficult
to understand
> ATM.  I think
> >> > >> >> >> what we need to do is think about what the responsibilities
> of the code are
> >> > >> >> >> and allow for different implementations that
are not so
> tightly coupled as
> >> > >> >> >> they are now.  For instance, having worked with
Storm to
> ingest millions of
> >> > >> >> >> activities a day, I personally would like to
see streams be
> responsible for
> >> > >> >> >> defining an over-arching orchestration model
that can be
> implemented within
> >> > >> >> >> a single war or on top of a distributed system.
 This would
> look something
> >> > >> >> >> like the follows:
> >> > >> >> >>
> >> > >> >> >> |__ Streams-Core  (Base classes, interfaces,
utilities,
> extensions, etc)
> >> > >> >> >> |
> >> > >> >> >> |__ Streams-Storm (Storm implementation of the
core)
> >> > >> >> >> |
> >> > >> >> >> |__ Streams-Camel (Camel implementation of the
core)
> >> > >> >> >> |
> >> > >> >> >> |__ Streams-WS (Web service implementation)
> >> > >> >> >>
> >> > >> >> >>
> >> > >> >> >> > Danny
> >> > >> >> >> >
> >> > >> >> >> >
> >> > >> >> >> > > From: dsullivan7@hotmail.com
> >> > >> >> >> > > To: dev@streams.incubator.apache.org
> >> > >> >> >> > > Subject: [DISCUSS] Switching Streams
from Camel
> deployment to .war
> >> > >> >> >> > deployment
> >> > >> >> >> > > Date: Tue, 29 Oct 2013 11:07:39 -0400
> >> > >> >> >> > >
> >> > >> >> >> > > The discussion thread for switching
Streams from
> Camel/osgi/Servicemix
> >> > >> >> >> > to a single .war deployment
> >> > >> >> >> >
> >> > >> >> >> >
> >> > >> >> >
> >> > >> >
> >> > >
> >>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message