streams-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matt Franklin <m.ben.frank...@gmail.com>
Subject Re: [DISCUSS] Switching Streams from Camel deployment to .war deployment
Date Thu, 31 Oct 2013 21:32:18 GMT
On Thu, Oct 31, 2013 at 5:28 PM, Chris Geer <chris@cxtsoftware.com> wrote:

> On Thu, Oct 31, 2013 at 2:23 PM, Matt Franklin <m.ben.franklin@gmail.com
> >wrote:
>
> > On Thu, Oct 31, 2013 at 5:04 PM, Danny Sullivan <dsullivan7@hotmail.com
> > >wrote:
> >
> > > I'm not quite following this, so I apologize. What I'm trying to do is
> > > programmatically make a request to a jar running on a separate jvm and
> > get
> > > the response from that call all within the same method. Similar to this
> > > http request:
> > >
> > > HttpGet httpget = new HttpGet();
> > >
> > > httpget.setURI(new URI("www.streams-persistence.com"));
> > >
> > > CloseableHttpResponse response = httpclient.execute(httpget);
> > >
> > > //do stuff with the response...
> > >
> > >
> > >
> > >
> > > I imagine that this would translate to something in Camel similar to
> > this:
> > >
> > >
> > > <route>
> > >
> > > <from uri="bean:subscriberService?method=getAllSubscribers"/>
> > >
> > > <inOut
> > >
> >
> uri="addressToStreamsPersistenceJar?method=selectAllSubscribersFromDatabase"/>
> > >
> > > </route>
> > >
> >
> > Camel uses AMQP as a messaging system so that you don't have to create
> HTTP
> > requests between sources, though it is possible to do so.  In Storm, we
> > usually try to use a buffer system like Kafka to do the same.
> >
>
> Camel can use AMQP through ActiveMQ.
>
>
Can being the operative word.  I haven't done much with Camel.  What other
transports are available.


> >
> >
> > >
> > >
> > > But it is unclear what the actual implementation would be.
> > >
> > >
> > > This actually brings me to another suggestion. Would there be a big
> > > performance impact to have communication between the software
> components
> > > occur between http? Say the 5 software components I outlined earlier
> were
> > > packaged as 5 separate wars. These wars could communicate with each
> other
> > > via get a post requests. This sounds unconventional offhand so I'd like
> > to
> > > hear some thoughts on it.
> > >
> >
> > It is most certainly possible, but IMO probably not the best option for
> > success.  Protocols like PubSubHubbub use HTTP for a message transport.
> >
> >
> > > -Danny
> > >
> > > > Date: Thu, 31 Oct 2013 13:28:41 -0700
> > > > Subject: Re: [DISCUSS] Switching Streams from Camel deployment to
> .war
> > > deployment
> > > > From: chris@cxtsoftware.com
> > > > To: dev@streams.incubator.apache.org
> > > >
> > > > Or Content Enricher [2]
> > > >
> > > > [2] http://camel.apache.org/content-enricher.html
> > > >
> > > >
> > > > On Thu, Oct 31, 2013 at 12:37 PM, Jason Letourneau
> > > > <jletourneau80@gmail.com>wrote:
> > > >
> > > > > check out the link here[1]
> > > > >
> > > > >
> > > > > [1]http://camel.apache.org/request-reply.html
> > > > >
> > > > > On Thu, Oct 31, 2013 at 2:32 PM, Danny Sullivan <
> > > dsullivan7@hotmail.com>
> > > > > wrote:
> > > > > > I have a quick Camel question that I arrived at in the
> > > implementation of
> > > > > these new components:
> > > > > >
> > > > > > Lets say I have a method in streams-activity.jar that needs
all
> > > > > subscribers in the database. This would require a call to the
> > > > > streams-persistence.jar. So far, I've seen camel used mostly for
> > > passing
> > > > > data through the application, but not for making a single
> > > request-reponse
> > > > > from within a method. How can I use Camel to get a list of all
> > > subscribers
> > > > > in the streams-persistence.jar from the streams-activity.jar?
> > > > > >
> > > > > >> From: dsullivan7@hotmail.com
> > > > > >> To: dev@streams.incubator.apache.org
> > > > > >> Subject: RE: [DISCUSS] Switching Streams from Camel deployment
> to
> > > .war
> > > > > deployment
> > > > > >> Date: Thu, 31 Oct 2013 08:40:44 -0400
> > > > > >>
> > > > > >> Excellent, I'll write up something as a proof of concept
and we
> > can
> > > > > discuss further to make sure everything is vanilla.
> > > > > >>
> > > > > >> > Date: Wed, 30 Oct 2013 17:01:28 -0400
> > > > > >> > Subject: Re: [DISCUSS] Switching Streams from Camel
deployment
> > to
> > > > > .war deployment
> > > > > >> > From: jletourneau80@gmail.com
> > > > > >> > To: dev@streams.incubator.apache.org
> > > > > >> >
> > > > > >> > That sounds pretty promising to me.
> > > > > >> >
> > > > > >> > On Wed, Oct 30, 2013 at 2:38 PM, Danny Sullivan <
> > > > > dsullivan7@hotmail.com> wrote:
> > > > > >> > > Thanks for the feedback. You have an interesting
point about
> > the
> > > > > url linking to a separate processing space. Let me tie my answer
> into
> > > your
> > > > > last question about "advocating for the simplicity at registration
> to
> > > give
> > > > > up flexibility at registration, but retaining the inner "guts" of
> > > > > EIP/messaging". Consider a new architecture:
> > > > > >> > >
> > > > > >> > > streams-web.war: single entry point to application,
but
> > > functions
> > > > > ONLY as an entry point. From here Camel routes the incoming
> requests
> > > to 4
> > > > > separate jarssubscriber-registration.jar: subscriber registration
> > > > > publisher-registration.jar: publisher registrationactivity.jar:
> > returns
> > > > > activity (also contains subscriber warehouse and storm activity
> > > > > aggregator)publish.jar: publishes activitystreams-cassandra.jar:
> the
> > > above
> > > > > 4 jars would all have a hook into this jar which would function as
> a
> > > hook
> > > > > onto the database. Each jar would have camel route output to this
> > jar.
> > > > > >> > >
> > > > > >> > > In this implementation, Camel would no longer
be the entry
> and
> > > exit
> > > > > point of a client to the application, but would handle the
> > > communication
> > > > > between components. The flow of activity through the application
> > would
> > > be
> > > > > method based in each jar. This would allow deployment on up to 6
> > > different
> > > > > process spaces. However, this does not address that there is a
> single
> > > > > server entry point, but I'm not sure if it was a concern in the
> first
> > > place.
> > > > > >> > >
> > > > > >> > > My argument, at its basis, is that we should move
away from
> > > using
> > > > > Camel as the entry point to the application. I would be happy to
> > > maintain
> > > > > messaging between components.
> > > > > >> > >
> > > > > >> > >> Date: Wed, 30 Oct 2013 12:32:55 -0400
> > > > > >> > >> Subject: Re: [DISCUSS] Switching Streams from
Camel
> > deployment
> > > to
> > > > > .war deployment
> > > > > >> > >> From: jletourneau80@gmail.com
> > > > > >> > >> To: dev@streams.incubator.apache.org
> > > > > >> > >>
> > > > > >> > >> An interesting use case that I am holding
onto is the
> ability
> > > for
> > > > > >> > >> publishers to register via a single URL (registration
> > > endpoint),
> > > > > but
> > > > > >> > >> be sent a URL back to post to a different
process space for
> > > actual
> > > > > >> > >> publishing.  The same is true on the subscriber
front.
> > >  Currently,
> > > > > the
> > > > > >> > >> Camel/EIP infrastructure abstracts this because
different
> > > > > components
> > > > > >> > >> deployed in different process spaces handling
the route
> > > creation
> > > > > can
> > > > > >> > >> just be bolted onto a running Streams instance
without new
> > > > > subs/pubs
> > > > > >> > >> behaving any differently than existing.  This
seems to be a
> > > > > >> > >> potentially critical scaling point.  Is there
a way to do
> > this
> > > with
> > > > > >> > >> the Spring solution?
> > > > > >> > >>
> > > > > >> > >> The persistence point is a good one, though
I would
> classify
> > > that
> > > > > as
> > > > > >> > >> "not implemented" vs "not possible" (not that
you were).
> > > > > >> > >>
> > > > > >> > >> I'm not married to Camel, I just like the
EIP approach to
> > > building
> > > > > >> > >> something that is ultimately a messaging system.
 There are
> > > known
> > > > > >> > >> patterns that solve at least a subset of the
problems
> Streams
> > > is
> > > > > >> > >> trying to solve and implementations that can
handle the
> load
> > > and
> > > > > I'll
> > > > > >> > >> reiterate flexibility == complexity almost
always.
> > > > > >> > >>
> > > > > >> > >> It comes right back to the central question:
Do you want
> > > > > flexibility
> > > > > >> > >> or simplicity?  It doesn't have to be black
and white
> either
> > I
> > > > > don't
> > > > > >> > >> think...
> > > > > >> > >>
> > > > > >> > >> More pointedly: Where should we give up flexibility
for
> > > > > simplicity?  I
> > > > > >> > >> read that Danny is advocating for the simplicity
at
> > > registration to
> > > > > >> > >> give up flexibility at registration, but retaining
the
> inner
> > > > > "guts" of
> > > > > >> > >> EIP/messaging?  Thoughts?
> > > > > >> > >>
> > > > > >> > >> On Wed, Oct 30, 2013 at 11:37 AM, Danny Sullivan
<
> > > > > dsullivan7@hotmail.com> wrote:
> > > > > >> > >> > My argument is not for the IoC pattern
as that can be
> (and
> > > has
> > > > > been) implemented alongside Camel. My main argument is that the
> > syntax
> > > at
> > > > > the entry point is not only familiar but much simpler. This
> wouldn't
> > > be a
> > > > > very strong argument if the Camel implementation wasn't much more
> > > > > complicated but I feel that it is the case. Also, looking toward
> the
> > > > > future, if the server is restarted, in-routes are lost in Camel.
> The
> > > way to
> > > > > curb this is to persist the dynamic routes that Camel creates, and
> > > then on
> > > > > start up pull every one of these routes and recreate a dynamic
> route
> > > for
> > > > > each one. Not only is this much easier to implement using the
> Spring
> > > web
> > > > > implementation, but it already has been implemented and you can try
> > it
> > > by
> > > > > checking out the webservice branch, registering a subscriber,
> > > restarting
> > > > > tomcat, and using the same url you had before. This will allow
> > > subscribers
> > > > > to hang on to their urls once they register. (the same is true for
> > > > > publishers: you can post via the same url after restarting tomcat)
> > > > > >> > >> >
> > > > > >> > >> >> Date: Wed, 30 Oct 2013 11:00:21 -0400
> > > > > >> > >> >> Subject: Re: [DISCUSS] Switching
Streams from Camel
> > > deployment
> > > > > to .war deployment
> > > > > >> > >> >> From: jletourneau80@gmail.com
> > > > > >> > >> >> To: dev@streams.incubator.apache.org
> > > > > >> > >> >>
> > > > > >> > >> >> To be fair, while the current implementation
is heavily
> > > > > camel-based,
> > > > > >> > >> >> all of the interfaces related to
Streams functionality
> are
> > > not.
> > > > >  The
> > > > > >> > >> >> current model maps to what Matt has
outlined in my
> > opinion,
> > > > > though
> > > > > >> > >> >> packing names etc. probably don't
follow that exact
> > pattern.
> > > > > >> > >> >>
> > > > > >> > >> >> With regards to the complexity and
different components
> in
> > > the
> > > > > >> > >> >> registration process, this was a
cut at the abstraction
> > > based
> > > > > on the
> > > > > >> > >> >> assumption that different implementations
may be plugged
> > in
> > > and
> > > > > in
> > > > > >> > >> >> fact may live on different processor
space (ie. a
> polling
> > > > > publisher vs
> > > > > >> > >> >> a push publisher may be instantiated
on different
> servers
> > > but
> > > > > the
> > > > > >> > >> >> registration URL is staticly defined).
> > > > > >> > >> >>
> > > > > >> > >> >> Is the main argument I am seeing
for Spring  the
> > > familiarity of
> > > > > its
> > > > > >> > >> >> IoC pattern implementation and syntax
at the entry
> point?
> > > > > >> > >> >>
> > > > > >> > >> >> On Wed, Oct 30, 2013 at 10:53 AM,
Danny Sullivan <
> > > > > dsullivan7@hotmail.com> wrote:
> > > > > >> > >> >> > Could you clarify whether the
same entry points would
> > > exist
> > > > > for the camel implementation of the core (implementing the
> "process"
> > > > > method/ using a DynammicRouteBuilder) or would the webservice be
> the
> > > sole
> > > > > entry point to Streams and after it enters would it hand it off to
> > > Camel?
> > > > > And what would be the entry point for the Storm implementation?
> > > > > >> > >> >> > -Danny
> > > > > >> > >> >> >
> > > > > >> > >> >> >> Date: Wed, 30 Oct 2013 10:13:04
-0400
> > > > > >> > >> >> >> Subject: Re: [DISCUSS] Switching
Streams from Camel
> > > > > deployment to .war deployment
> > > > > >> > >> >> >> From: m.ben.franklin@gmail.com
> > > > > >> > >> >> >> To: dev@streams.incubator.apache.org
> > > > > >> > >> >> >>
> > > > > >> > >> >> >> On Tue, Oct 29, 2013 at
2:55 PM, Danny Sullivan <
> > > > > dsullivan7@hotmail.com>wrote:
> > > > > >> > >> >> >>
> > > > > >> > >> >> >> > My case for switching
from OSGi is for simplicity
> in
> > > > > design. To follow the
> > > > > >> > >> >> >> > path of an activity
through Streams in the
> > webservice,
> > > > > there is one main
> > > > > >> > >> >> >> > things the developer
needs to understand:
> > > > > >> > >> >> >> > The @RequestMapping
annotation specifies the HTTP
> > entry
> > > > > point
> > > > > >> > >> >> >> > There are 4 @RequestMapping
annotations that
> > > correspond to
> > > > > each of the 4
> > > > > >> > >> >> >> > ways a user enters
the application: registering a
> > > > > publisher, registering a
> > > > > >> > >> >> >> > subscriber, publishing
activity, and getting an
> > > activity
> > > > > stream. Where
> > > > > >> > >> >> >> > these are located in
the source code can be found
> by
> > > > > searching for the
> > > > > >> > >> >> >> > paths specified in
the documentation (search for
> > > > > "/publisherRegister",
> > > > > >> > >> >> >> > "/subscriberRegister",
"/publishActivity",
> > > "/getActivity"
> > > > > which will all
> > > > > >> > >> >> >> > lead you to StreamsWebController.java).
From the
> > > methods
> > > > > that process
> > > > > >> > >> >> >> > requests, the flow
through the application is
> through
> > > > > methods which can be
> > > > > >> > >> >> >> > understood by most
Java programmers.
> > > > > >> > >> >> >> > The flow of activities
through the current trunk
> > > branch is
> > > > > understood as
> > > > > >> > >> >> >> > follows:
> > > > > >> > >> >> >> > The string "/publisher/register"
(the entry point
> to
> > > > > register a publisher
> > > > > >> > >> >> >> > specified in the documentation)
is the value of the
> > > > > >> > >> >> >> > consumer.registrationEndpoint
property defined in
> > > > > streams.propertiesThe
> > > > > >> > >> >> >> > camelContext.xml specifies
an endpoint with the id
> > > > > >> > >> >> >> > consumerRegistrationEndpoint
to have a uri equal to
> > the
> > > > > propertyThe
> > > > > >> > >> >> >> > consumerRegistrationEndpoint
routes from uri
> > > > > direct:publisher register with
> > > > > >> > >> >> >> > the bean activityRegistrationProcessor
nested in
> > > between
> > > > > the routeThe
> > > > > >> > >> >> >> > streams-eip-applicationContext
contains a bean with
> > > the id
> > > > > >> > >> >> >> > activityRegistrationProcessor
created for the class
> > > > > >> > >> >> >> > ActivityPublisherRegistrationProcessorThe
exchange
> > will
> > > > > enter the "process"
> > > > > >> > >> >> >> > method of the class
> > > > > ActivityPublisherRegisitrationProcessor and that this
> > > > > >> > >> >> >> > is because this class
implements the "Processor"
> > > interface
> > > > > provided by
> > > > > >> > >> >> >> > CamelThe direct:add-publisher-route
takes the
> > exchange
> > > > > output from the
> > > > > >> > >> >> >> > "process" method and
routes it to the
> > > > > activityRegistrationProcessor
> > > > > >> > >> >> >> > "register" method.
The bean
> > > activityRegistrationProcessor
> > > > > is defined in the
> > > > > >> > >> >> >> > streams-eip-osgi-component-import.xmlThe
output
> from
> > > this
> > > > > method is then
> > > > > >> > >> >> >> > sent to the "createNewRouteForConsumer"
method of
> > > > > activityConsumerRouter.
> > > > > >> > >> >> >> > This method creates
a new route for the newly
> > > registered
> > > > > publisher using
> > > > > >> > >> >> >> > the private static
final class
> > > DynamicConsumerRouteBuilder
> > > > > which is
> > > > > >> > >> >> >> > required to extend
RouteBuilder which is provided
> by
> > > > > Camel. This
> > > > > >> > >> >> >> > DynamicConsumerRouteBuilder
contains several
> methods:
> > > > > >> > >> >> >> > getConsumerReceiveMethod()
(which corresponds to
> > @Value
> > > > > >> > >> >> >> > ${consumer.receiveMethod}
which corresponds to
> > > "receive"),
> > > > > >> > >> >> >> > getConsumerSplitMethod()
(@Value
> > > > > ${consumer.splitMethod},"split"), and
> > > > > >> > >> >> >> > getConsumerActivityQUri()
(@Value
> > > ${consumerActivityQUri},
> > > > > >> > >> >> >> > "direct:activityQ").
This is different than the
> > > > > camelContext.xml in that
> > > > > >> > >> >> >> > the route is being
created programmatically. What
> > this
> > > is
> > > > > doing is routing
> > > > > >> > >> >> >> > input from the inroute
url (which Camel does
> > > automatically
> > > > > through the
> > > > > >> > >> >> >> > configure method which
is required to be
> overridden),
> > > to
> > > > > the "receive"
> > > > > >> > >> >> >> > method of ActivityConsumer,
then to the "split"
> > method
> > > of
> > > > > ActivityConsumer,
> > > > > >> > >> >> >> > and then to the "direct:activityQ"
which if you
> look
> > > back
> > > > > in the
> > > > > >> > >> >> >> > camelContext.xml routes
to the
> > > "activemq:queue:activities"
> > > > > which then
> > > > > >> > >> >> >> > routes to "receiveExchange"
> > > > > >> > >> >> >> > This is the process
to register a publisher. The
> > > process
> > > > > for registering a
> > > > > >> > >> >> >> > subscriber is relatively
the same though it
> involves
> > > > > separate classes with
> > > > > >> > >> >> >> > their own private static
final RouteBuilder class.
> > > From my
> > > > > perspective, the
> > > > > >> > >> >> >> > two most difficult
things with setting this project
> > up
> > > > > were understanding
> > > > > >> > >> >> >> > that the "process"
method of the class that
> > implements
> > > > > "Processor" is the
> > > > > >> > >> >> >> > entry point and the
DynamicConsumerRouteBuilder
> > creates
> > > > > the second entry
> > > > > >> > >> >> >> > point (The 5th and
last points). This made the
> > project
> > > > > very, VERY hard to
> > > > > >> > >> >> >> > understand.
> > > > > >> > >> >> >> > In addition to simplicity
of design, the mvn clean
> > > install
> > > > > of the web
> > > > > >> > >> >> >> > service project is
much faster and small scale
> > activity
> > > > > publishing is also
> > > > > >> > >> >> >> > faster (see my email
about load testing). These are
> > > minor
> > > > > points though as
> > > > > >> > >> >> >> > compilation has no
effect on deployment. OSGi does
> > add
> > > the
> > > > > benefit of
> > > > > >> > >> >> >> > modularized programming
which is valuable, though I
> > > think
> > > > > the added
> > > > > >> > >> >> >> > complexity of Camel
merits moving the project away
> > from
> > > > > this paradigm.
> > > > > >> > >> >> >> >
> > > > > >> > >> >> >>
> > > > > >> > >> >> >> I agree that the project
is pretty difficult to
> > > understand
> > > > > ATM.  I think
> > > > > >> > >> >> >> what we need to do is think
about what the
> > > responsibilities
> > > > > of the code are
> > > > > >> > >> >> >> and allow for different
implementations that are not
> so
> > > > > tightly coupled as
> > > > > >> > >> >> >> they are now.  For instance,
having worked with Storm
> > to
> > > > > ingest millions of
> > > > > >> > >> >> >> activities a day, I personally
would like to see
> > streams
> > > be
> > > > > responsible for
> > > > > >> > >> >> >> defining an over-arching
orchestration model that can
> > be
> > > > > implemented within
> > > > > >> > >> >> >> a single war or on top of
a distributed system.  This
> > > would
> > > > > look something
> > > > > >> > >> >> >> like the follows:
> > > > > >> > >> >> >>
> > > > > >> > >> >> >> |__ Streams-Core  (Base
classes, interfaces,
> utilities,
> > > > > extensions, etc)
> > > > > >> > >> >> >> |
> > > > > >> > >> >> >> |__ Streams-Storm (Storm
implementation of the core)
> > > > > >> > >> >> >> |
> > > > > >> > >> >> >> |__ Streams-Camel (Camel
implementation of the core)
> > > > > >> > >> >> >> |
> > > > > >> > >> >> >> |__ Streams-WS (Web service
implementation)
> > > > > >> > >> >> >>
> > > > > >> > >> >> >>
> > > > > >> > >> >> >> > Danny
> > > > > >> > >> >> >> >
> > > > > >> > >> >> >> >
> > > > > >> > >> >> >> > > From: dsullivan7@hotmail.com
> > > > > >> > >> >> >> > > To: dev@streams.incubator.apache.org
> > > > > >> > >> >> >> > > Subject: [DISCUSS]
Switching Streams from Camel
> > > > > deployment to .war
> > > > > >> > >> >> >> > deployment
> > > > > >> > >> >> >> > > Date: Tue, 29
Oct 2013 11:07:39 -0400
> > > > > >> > >> >> >> > >
> > > > > >> > >> >> >> > > The discussion
thread for switching Streams from
> > > > > Camel/osgi/Servicemix
> > > > > >> > >> >> >> > to a single .war deployment
> > > > > >> > >> >> >> >
> > > > > >> > >> >> >> >
> > > > > >> > >> >> >
> > > > > >> > >> >
> > > > > >> > >
> > > > > >>
> > > > > >
> > > > >
> > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message