streams-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matt Franklin <m.ben.frank...@gmail.com>
Subject Re: [DISCUSS] Switching Streams from Camel deployment to .war deployment
Date Thu, 31 Oct 2013 21:23:20 GMT
On Thu, Oct 31, 2013 at 5:04 PM, Danny Sullivan <dsullivan7@hotmail.com>wrote:

> I'm not quite following this, so I apologize. What I'm trying to do is
> programmatically make a request to a jar running on a separate jvm and get
> the response from that call all within the same method. Similar to this
> http request:
>
> HttpGet httpget = new HttpGet();
>
> httpget.setURI(new URI("www.streams-persistence.com"));
>
> CloseableHttpResponse response = httpclient.execute(httpget);
>
> //do stuff with the response...
>
>
>
>
> I imagine that this would translate to something in Camel similar to this:
>
>
> <route>
>
> <from uri="bean:subscriberService?method=getAllSubscribers"/>
>
> <inOut
> uri="addressToStreamsPersistenceJar?method=selectAllSubscribersFromDatabase"/>
>
> </route>
>

Camel uses AMQP as a messaging system so that you don't have to create HTTP
requests between sources, though it is possible to do so.  In Storm, we
usually try to use a buffer system like Kafka to do the same.


>
>
> But it is unclear what the actual implementation would be.
>
>
> This actually brings me to another suggestion. Would there be a big
> performance impact to have communication between the software components
> occur between http? Say the 5 software components I outlined earlier were
> packaged as 5 separate wars. These wars could communicate with each other
> via get a post requests. This sounds unconventional offhand so I'd like to
> hear some thoughts on it.
>

It is most certainly possible, but IMO probably not the best option for
success.  Protocols like PubSubHubbub use HTTP for a message transport.


> -Danny
>
> > Date: Thu, 31 Oct 2013 13:28:41 -0700
> > Subject: Re: [DISCUSS] Switching Streams from Camel deployment to .war
> deployment
> > From: chris@cxtsoftware.com
> > To: dev@streams.incubator.apache.org
> >
> > Or Content Enricher [2]
> >
> > [2] http://camel.apache.org/content-enricher.html
> >
> >
> > On Thu, Oct 31, 2013 at 12:37 PM, Jason Letourneau
> > <jletourneau80@gmail.com>wrote:
> >
> > > check out the link here[1]
> > >
> > >
> > > [1]http://camel.apache.org/request-reply.html
> > >
> > > On Thu, Oct 31, 2013 at 2:32 PM, Danny Sullivan <
> dsullivan7@hotmail.com>
> > > wrote:
> > > > I have a quick Camel question that I arrived at in the
> implementation of
> > > these new components:
> > > >
> > > > Lets say I have a method in streams-activity.jar that needs all
> > > subscribers in the database. This would require a call to the
> > > streams-persistence.jar. So far, I've seen camel used mostly for
> passing
> > > data through the application, but not for making a single
> request-reponse
> > > from within a method. How can I use Camel to get a list of all
> subscribers
> > > in the streams-persistence.jar from the streams-activity.jar?
> > > >
> > > >> From: dsullivan7@hotmail.com
> > > >> To: dev@streams.incubator.apache.org
> > > >> Subject: RE: [DISCUSS] Switching Streams from Camel deployment to
> .war
> > > deployment
> > > >> Date: Thu, 31 Oct 2013 08:40:44 -0400
> > > >>
> > > >> Excellent, I'll write up something as a proof of concept and we can
> > > discuss further to make sure everything is vanilla.
> > > >>
> > > >> > Date: Wed, 30 Oct 2013 17:01:28 -0400
> > > >> > Subject: Re: [DISCUSS] Switching Streams from Camel deployment
to
> > > .war deployment
> > > >> > From: jletourneau80@gmail.com
> > > >> > To: dev@streams.incubator.apache.org
> > > >> >
> > > >> > That sounds pretty promising to me.
> > > >> >
> > > >> > On Wed, Oct 30, 2013 at 2:38 PM, Danny Sullivan <
> > > dsullivan7@hotmail.com> wrote:
> > > >> > > Thanks for the feedback. You have an interesting point about
the
> > > url linking to a separate processing space. Let me tie my answer into
> your
> > > last question about "advocating for the simplicity at registration to
> give
> > > up flexibility at registration, but retaining the inner "guts" of
> > > EIP/messaging". Consider a new architecture:
> > > >> > >
> > > >> > > streams-web.war: single entry point to application, but
> functions
> > > ONLY as an entry point. From here Camel routes the incoming requests
> to 4
> > > separate jarssubscriber-registration.jar: subscriber registration
> > > publisher-registration.jar: publisher registrationactivity.jar: returns
> > > activity (also contains subscriber warehouse and storm activity
> > > aggregator)publish.jar: publishes activitystreams-cassandra.jar: the
> above
> > > 4 jars would all have a hook into this jar which would function as a
> hook
> > > onto the database. Each jar would have camel route output to this jar.
> > > >> > >
> > > >> > > In this implementation, Camel would no longer be the entry
and
> exit
> > > point of a client to the application, but would handle the
> communication
> > > between components. The flow of activity through the application would
> be
> > > method based in each jar. This would allow deployment on up to 6
> different
> > > process spaces. However, this does not address that there is a single
> > > server entry point, but I'm not sure if it was a concern in the first
> place.
> > > >> > >
> > > >> > > My argument, at its basis, is that we should move away from
> using
> > > Camel as the entry point to the application. I would be happy to
> maintain
> > > messaging between components.
> > > >> > >
> > > >> > >> Date: Wed, 30 Oct 2013 12:32:55 -0400
> > > >> > >> Subject: Re: [DISCUSS] Switching Streams from Camel
deployment
> to
> > > .war deployment
> > > >> > >> From: jletourneau80@gmail.com
> > > >> > >> To: dev@streams.incubator.apache.org
> > > >> > >>
> > > >> > >> An interesting use case that I am holding onto is the
ability
> for
> > > >> > >> publishers to register via a single URL (registration
> endpoint),
> > > but
> > > >> > >> be sent a URL back to post to a different process space
for
> actual
> > > >> > >> publishing.  The same is true on the subscriber front.
>  Currently,
> > > the
> > > >> > >> Camel/EIP infrastructure abstracts this because different
> > > components
> > > >> > >> deployed in different process spaces handling the route
> creation
> > > can
> > > >> > >> just be bolted onto a running Streams instance without
new
> > > subs/pubs
> > > >> > >> behaving any differently than existing.  This seems
to be a
> > > >> > >> potentially critical scaling point.  Is there a way
to do this
> with
> > > >> > >> the Spring solution?
> > > >> > >>
> > > >> > >> The persistence point is a good one, though I would
classify
> that
> > > as
> > > >> > >> "not implemented" vs "not possible" (not that you were).
> > > >> > >>
> > > >> > >> I'm not married to Camel, I just like the EIP approach
to
> building
> > > >> > >> something that is ultimately a messaging system.  There
are
> known
> > > >> > >> patterns that solve at least a subset of the problems
Streams
> is
> > > >> > >> trying to solve and implementations that can handle
the load
> and
> > > I'll
> > > >> > >> reiterate flexibility == complexity almost always.
> > > >> > >>
> > > >> > >> It comes right back to the central question: Do you
want
> > > flexibility
> > > >> > >> or simplicity?  It doesn't have to be black and white
either I
> > > don't
> > > >> > >> think...
> > > >> > >>
> > > >> > >> More pointedly: Where should we give up flexibility
for
> > > simplicity?  I
> > > >> > >> read that Danny is advocating for the simplicity at
> registration to
> > > >> > >> give up flexibility at registration, but retaining the
inner
> > > "guts" of
> > > >> > >> EIP/messaging?  Thoughts?
> > > >> > >>
> > > >> > >> On Wed, Oct 30, 2013 at 11:37 AM, Danny Sullivan <
> > > dsullivan7@hotmail.com> wrote:
> > > >> > >> > My argument is not for the IoC pattern as that
can be (and
> has
> > > been) implemented alongside Camel. My main argument is that the syntax
> at
> > > the entry point is not only familiar but much simpler. This wouldn't
> be a
> > > very strong argument if the Camel implementation wasn't much more
> > > complicated but I feel that it is the case. Also, looking toward the
> > > future, if the server is restarted, in-routes are lost in Camel. The
> way to
> > > curb this is to persist the dynamic routes that Camel creates, and
> then on
> > > start up pull every one of these routes and recreate a dynamic route
> for
> > > each one. Not only is this much easier to implement using the Spring
> web
> > > implementation, but it already has been implemented and you can try it
> by
> > > checking out the webservice branch, registering a subscriber,
> restarting
> > > tomcat, and using the same url you had before. This will allow
> subscribers
> > > to hang on to their urls once they register. (the same is true for
> > > publishers: you can post via the same url after restarting tomcat)
> > > >> > >> >
> > > >> > >> >> Date: Wed, 30 Oct 2013 11:00:21 -0400
> > > >> > >> >> Subject: Re: [DISCUSS] Switching Streams from
Camel
> deployment
> > > to .war deployment
> > > >> > >> >> From: jletourneau80@gmail.com
> > > >> > >> >> To: dev@streams.incubator.apache.org
> > > >> > >> >>
> > > >> > >> >> To be fair, while the current implementation
is heavily
> > > camel-based,
> > > >> > >> >> all of the interfaces related to Streams functionality
are
> not.
> > >  The
> > > >> > >> >> current model maps to what Matt has outlined
in my opinion,
> > > though
> > > >> > >> >> packing names etc. probably don't follow that
exact pattern.
> > > >> > >> >>
> > > >> > >> >> With regards to the complexity and different
components in
> the
> > > >> > >> >> registration process, this was a cut at the
abstraction
> based
> > > on the
> > > >> > >> >> assumption that different implementations may
be plugged in
> and
> > > in
> > > >> > >> >> fact may live on different processor space
(ie. a polling
> > > publisher vs
> > > >> > >> >> a push publisher may be instantiated on different
servers
> but
> > > the
> > > >> > >> >> registration URL is staticly defined).
> > > >> > >> >>
> > > >> > >> >> Is the main argument I am seeing for Spring
 the
> familiarity of
> > > its
> > > >> > >> >> IoC pattern implementation and syntax at the
entry point?
> > > >> > >> >>
> > > >> > >> >> On Wed, Oct 30, 2013 at 10:53 AM, Danny Sullivan
<
> > > dsullivan7@hotmail.com> wrote:
> > > >> > >> >> > Could you clarify whether the same entry
points would
> exist
> > > for the camel implementation of the core (implementing the "process"
> > > method/ using a DynammicRouteBuilder) or would the webservice be the
> sole
> > > entry point to Streams and after it enters would it hand it off to
> Camel?
> > > And what would be the entry point for the Storm implementation?
> > > >> > >> >> > -Danny
> > > >> > >> >> >
> > > >> > >> >> >> Date: Wed, 30 Oct 2013 10:13:04 -0400
> > > >> > >> >> >> Subject: Re: [DISCUSS] Switching Streams
from Camel
> > > deployment to .war deployment
> > > >> > >> >> >> From: m.ben.franklin@gmail.com
> > > >> > >> >> >> To: dev@streams.incubator.apache.org
> > > >> > >> >> >>
> > > >> > >> >> >> On Tue, Oct 29, 2013 at 2:55 PM, Danny
Sullivan <
> > > dsullivan7@hotmail.com>wrote:
> > > >> > >> >> >>
> > > >> > >> >> >> > My case for switching from OSGi
is for simplicity in
> > > design. To follow the
> > > >> > >> >> >> > path of an activity through Streams
in the webservice,
> > > there is one main
> > > >> > >> >> >> > things the developer needs to
understand:
> > > >> > >> >> >> > The @RequestMapping annotation
specifies the HTTP entry
> > > point
> > > >> > >> >> >> > There are 4 @RequestMapping annotations
that
> correspond to
> > > each of the 4
> > > >> > >> >> >> > ways a user enters the application:
registering a
> > > publisher, registering a
> > > >> > >> >> >> > subscriber, publishing activity,
and getting an
> activity
> > > stream. Where
> > > >> > >> >> >> > these are located in the source
code can be found by
> > > searching for the
> > > >> > >> >> >> > paths specified in the documentation
(search for
> > > "/publisherRegister",
> > > >> > >> >> >> > "/subscriberRegister", "/publishActivity",
> "/getActivity"
> > > which will all
> > > >> > >> >> >> > lead you to StreamsWebController.java).
From the
> methods
> > > that process
> > > >> > >> >> >> > requests, the flow through the
application is through
> > > methods which can be
> > > >> > >> >> >> > understood by most Java programmers.
> > > >> > >> >> >> > The flow of activities through
the current trunk
> branch is
> > > understood as
> > > >> > >> >> >> > follows:
> > > >> > >> >> >> > The string "/publisher/register"
(the entry point to
> > > register a publisher
> > > >> > >> >> >> > specified in the documentation)
is the value of the
> > > >> > >> >> >> > consumer.registrationEndpoint
property defined in
> > > streams.propertiesThe
> > > >> > >> >> >> > camelContext.xml specifies an
endpoint with the id
> > > >> > >> >> >> > consumerRegistrationEndpoint
to have a uri equal to the
> > > propertyThe
> > > >> > >> >> >> > consumerRegistrationEndpoint
routes from uri
> > > direct:publisher register with
> > > >> > >> >> >> > the bean activityRegistrationProcessor
nested in
> between
> > > the routeThe
> > > >> > >> >> >> > streams-eip-applicationContext
contains a bean with
> the id
> > > >> > >> >> >> > activityRegistrationProcessor
created for the class
> > > >> > >> >> >> > ActivityPublisherRegistrationProcessorThe
exchange will
> > > enter the "process"
> > > >> > >> >> >> > method of the class
> > > ActivityPublisherRegisitrationProcessor and that this
> > > >> > >> >> >> > is because this class implements
the "Processor"
> interface
> > > provided by
> > > >> > >> >> >> > CamelThe direct:add-publisher-route
takes the exchange
> > > output from the
> > > >> > >> >> >> > "process" method and routes it
to the
> > > activityRegistrationProcessor
> > > >> > >> >> >> > "register" method. The bean
> activityRegistrationProcessor
> > > is defined in the
> > > >> > >> >> >> > streams-eip-osgi-component-import.xmlThe
output from
> this
> > > method is then
> > > >> > >> >> >> > sent to the "createNewRouteForConsumer"
method of
> > > activityConsumerRouter.
> > > >> > >> >> >> > This method creates a new route
for the newly
> registered
> > > publisher using
> > > >> > >> >> >> > the private static final class
> DynamicConsumerRouteBuilder
> > > which is
> > > >> > >> >> >> > required to extend RouteBuilder
which is provided by
> > > Camel. This
> > > >> > >> >> >> > DynamicConsumerRouteBuilder contains
several methods:
> > > >> > >> >> >> > getConsumerReceiveMethod() (which
corresponds to @Value
> > > >> > >> >> >> > ${consumer.receiveMethod} which
corresponds to
> "receive"),
> > > >> > >> >> >> > getConsumerSplitMethod() (@Value
> > > ${consumer.splitMethod},"split"), and
> > > >> > >> >> >> > getConsumerActivityQUri() (@Value
> ${consumerActivityQUri},
> > > >> > >> >> >> > "direct:activityQ"). This is
different than the
> > > camelContext.xml in that
> > > >> > >> >> >> > the route is being created programmatically.
What this
> is
> > > doing is routing
> > > >> > >> >> >> > input from the inroute url (which
Camel does
> automatically
> > > through the
> > > >> > >> >> >> > configure method which is required
to be overridden),
> to
> > > the "receive"
> > > >> > >> >> >> > method of ActivityConsumer, then
to the "split" method
> of
> > > ActivityConsumer,
> > > >> > >> >> >> > and then to the "direct:activityQ"
which if you look
> back
> > > in the
> > > >> > >> >> >> > camelContext.xml routes to the
> "activemq:queue:activities"
> > > which then
> > > >> > >> >> >> > routes to "receiveExchange"
> > > >> > >> >> >> > This is the process to register
a publisher. The
> process
> > > for registering a
> > > >> > >> >> >> > subscriber is relatively the
same though it involves
> > > separate classes with
> > > >> > >> >> >> > their own private static final
RouteBuilder class.
> From my
> > > perspective, the
> > > >> > >> >> >> > two most difficult things with
setting this project up
> > > were understanding
> > > >> > >> >> >> > that the "process" method of
the class that implements
> > > "Processor" is the
> > > >> > >> >> >> > entry point and the DynamicConsumerRouteBuilder
creates
> > > the second entry
> > > >> > >> >> >> > point (The 5th and last points).
This made the project
> > > very, VERY hard to
> > > >> > >> >> >> > understand.
> > > >> > >> >> >> > In addition to simplicity of
design, the mvn clean
> install
> > > of the web
> > > >> > >> >> >> > service project is much faster
and small scale activity
> > > publishing is also
> > > >> > >> >> >> > faster (see my email about load
testing). These are
> minor
> > > points though as
> > > >> > >> >> >> > compilation has no effect on
deployment. OSGi does add
> the
> > > benefit of
> > > >> > >> >> >> > modularized programming which
is valuable, though I
> think
> > > the added
> > > >> > >> >> >> > complexity of Camel merits moving
the project away from
> > > this paradigm.
> > > >> > >> >> >> >
> > > >> > >> >> >>
> > > >> > >> >> >> I agree that the project is pretty
difficult to
> understand
> > > ATM.  I think
> > > >> > >> >> >> what we need to do is think about
what the
> responsibilities
> > > of the code are
> > > >> > >> >> >> and allow for different implementations
that are not so
> > > tightly coupled as
> > > >> > >> >> >> they are now.  For instance, having
worked with Storm to
> > > ingest millions of
> > > >> > >> >> >> activities a day, I personally would
like to see streams
> be
> > > responsible for
> > > >> > >> >> >> defining an over-arching orchestration
model that can be
> > > implemented within
> > > >> > >> >> >> a single war or on top of a distributed
system.  This
> would
> > > look something
> > > >> > >> >> >> like the follows:
> > > >> > >> >> >>
> > > >> > >> >> >> |__ Streams-Core  (Base classes, interfaces,
utilities,
> > > extensions, etc)
> > > >> > >> >> >> |
> > > >> > >> >> >> |__ Streams-Storm (Storm implementation
of the core)
> > > >> > >> >> >> |
> > > >> > >> >> >> |__ Streams-Camel (Camel implementation
of the core)
> > > >> > >> >> >> |
> > > >> > >> >> >> |__ Streams-WS (Web service implementation)
> > > >> > >> >> >>
> > > >> > >> >> >>
> > > >> > >> >> >> > Danny
> > > >> > >> >> >> >
> > > >> > >> >> >> >
> > > >> > >> >> >> > > From: dsullivan7@hotmail.com
> > > >> > >> >> >> > > To: dev@streams.incubator.apache.org
> > > >> > >> >> >> > > Subject: [DISCUSS] Switching
Streams from Camel
> > > deployment to .war
> > > >> > >> >> >> > deployment
> > > >> > >> >> >> > > Date: Tue, 29 Oct 2013 11:07:39
-0400
> > > >> > >> >> >> > >
> > > >> > >> >> >> > > The discussion thread for
switching Streams from
> > > Camel/osgi/Servicemix
> > > >> > >> >> >> > to a single .war deployment
> > > >> > >> >> >> >
> > > >> > >> >> >> >
> > > >> > >> >> >
> > > >> > >> >
> > > >> > >
> > > >>
> > > >
> > >
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message