streams-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Geer <ch...@cxtsoftware.com>
Subject Re: [DISCUSS] Switching Streams from Camel deployment to .war deployment
Date Thu, 31 Oct 2013 21:28:31 GMT
On Thu, Oct 31, 2013 at 2:23 PM, Matt Franklin <m.ben.franklin@gmail.com>wrote:

> On Thu, Oct 31, 2013 at 5:04 PM, Danny Sullivan <dsullivan7@hotmail.com
> >wrote:
>
> > I'm not quite following this, so I apologize. What I'm trying to do is
> > programmatically make a request to a jar running on a separate jvm and
> get
> > the response from that call all within the same method. Similar to this
> > http request:
> >
> > HttpGet httpget = new HttpGet();
> >
> > httpget.setURI(new URI("www.streams-persistence.com"));
> >
> > CloseableHttpResponse response = httpclient.execute(httpget);
> >
> > //do stuff with the response...
> >
> >
> >
> >
> > I imagine that this would translate to something in Camel similar to
> this:
> >
> >
> > <route>
> >
> > <from uri="bean:subscriberService?method=getAllSubscribers"/>
> >
> > <inOut
> >
> uri="addressToStreamsPersistenceJar?method=selectAllSubscribersFromDatabase"/>
> >
> > </route>
> >
>
> Camel uses AMQP as a messaging system so that you don't have to create HTTP
> requests between sources, though it is possible to do so.  In Storm, we
> usually try to use a buffer system like Kafka to do the same.
>

Camel can use AMQP through ActiveMQ.

>
>
> >
> >
> > But it is unclear what the actual implementation would be.
> >
> >
> > This actually brings me to another suggestion. Would there be a big
> > performance impact to have communication between the software components
> > occur between http? Say the 5 software components I outlined earlier were
> > packaged as 5 separate wars. These wars could communicate with each other
> > via get a post requests. This sounds unconventional offhand so I'd like
> to
> > hear some thoughts on it.
> >
>
> It is most certainly possible, but IMO probably not the best option for
> success.  Protocols like PubSubHubbub use HTTP for a message transport.
>
>
> > -Danny
> >
> > > Date: Thu, 31 Oct 2013 13:28:41 -0700
> > > Subject: Re: [DISCUSS] Switching Streams from Camel deployment to .war
> > deployment
> > > From: chris@cxtsoftware.com
> > > To: dev@streams.incubator.apache.org
> > >
> > > Or Content Enricher [2]
> > >
> > > [2] http://camel.apache.org/content-enricher.html
> > >
> > >
> > > On Thu, Oct 31, 2013 at 12:37 PM, Jason Letourneau
> > > <jletourneau80@gmail.com>wrote:
> > >
> > > > check out the link here[1]
> > > >
> > > >
> > > > [1]http://camel.apache.org/request-reply.html
> > > >
> > > > On Thu, Oct 31, 2013 at 2:32 PM, Danny Sullivan <
> > dsullivan7@hotmail.com>
> > > > wrote:
> > > > > I have a quick Camel question that I arrived at in the
> > implementation of
> > > > these new components:
> > > > >
> > > > > Lets say I have a method in streams-activity.jar that needs all
> > > > subscribers in the database. This would require a call to the
> > > > streams-persistence.jar. So far, I've seen camel used mostly for
> > passing
> > > > data through the application, but not for making a single
> > request-reponse
> > > > from within a method. How can I use Camel to get a list of all
> > subscribers
> > > > in the streams-persistence.jar from the streams-activity.jar?
> > > > >
> > > > >> From: dsullivan7@hotmail.com
> > > > >> To: dev@streams.incubator.apache.org
> > > > >> Subject: RE: [DISCUSS] Switching Streams from Camel deployment
to
> > .war
> > > > deployment
> > > > >> Date: Thu, 31 Oct 2013 08:40:44 -0400
> > > > >>
> > > > >> Excellent, I'll write up something as a proof of concept and
we
> can
> > > > discuss further to make sure everything is vanilla.
> > > > >>
> > > > >> > Date: Wed, 30 Oct 2013 17:01:28 -0400
> > > > >> > Subject: Re: [DISCUSS] Switching Streams from Camel deployment
> to
> > > > .war deployment
> > > > >> > From: jletourneau80@gmail.com
> > > > >> > To: dev@streams.incubator.apache.org
> > > > >> >
> > > > >> > That sounds pretty promising to me.
> > > > >> >
> > > > >> > On Wed, Oct 30, 2013 at 2:38 PM, Danny Sullivan <
> > > > dsullivan7@hotmail.com> wrote:
> > > > >> > > Thanks for the feedback. You have an interesting point
about
> the
> > > > url linking to a separate processing space. Let me tie my answer into
> > your
> > > > last question about "advocating for the simplicity at registration to
> > give
> > > > up flexibility at registration, but retaining the inner "guts" of
> > > > EIP/messaging". Consider a new architecture:
> > > > >> > >
> > > > >> > > streams-web.war: single entry point to application,
but
> > functions
> > > > ONLY as an entry point. From here Camel routes the incoming requests
> > to 4
> > > > separate jarssubscriber-registration.jar: subscriber registration
> > > > publisher-registration.jar: publisher registrationactivity.jar:
> returns
> > > > activity (also contains subscriber warehouse and storm activity
> > > > aggregator)publish.jar: publishes activitystreams-cassandra.jar: the
> > above
> > > > 4 jars would all have a hook into this jar which would function as a
> > hook
> > > > onto the database. Each jar would have camel route output to this
> jar.
> > > > >> > >
> > > > >> > > In this implementation, Camel would no longer be the
entry and
> > exit
> > > > point of a client to the application, but would handle the
> > communication
> > > > between components. The flow of activity through the application
> would
> > be
> > > > method based in each jar. This would allow deployment on up to 6
> > different
> > > > process spaces. However, this does not address that there is a single
> > > > server entry point, but I'm not sure if it was a concern in the first
> > place.
> > > > >> > >
> > > > >> > > My argument, at its basis, is that we should move away
from
> > using
> > > > Camel as the entry point to the application. I would be happy to
> > maintain
> > > > messaging between components.
> > > > >> > >
> > > > >> > >> Date: Wed, 30 Oct 2013 12:32:55 -0400
> > > > >> > >> Subject: Re: [DISCUSS] Switching Streams from Camel
> deployment
> > to
> > > > .war deployment
> > > > >> > >> From: jletourneau80@gmail.com
> > > > >> > >> To: dev@streams.incubator.apache.org
> > > > >> > >>
> > > > >> > >> An interesting use case that I am holding onto
is the ability
> > for
> > > > >> > >> publishers to register via a single URL (registration
> > endpoint),
> > > > but
> > > > >> > >> be sent a URL back to post to a different process
space for
> > actual
> > > > >> > >> publishing.  The same is true on the subscriber
front.
> >  Currently,
> > > > the
> > > > >> > >> Camel/EIP infrastructure abstracts this because
different
> > > > components
> > > > >> > >> deployed in different process spaces handling the
route
> > creation
> > > > can
> > > > >> > >> just be bolted onto a running Streams instance
without new
> > > > subs/pubs
> > > > >> > >> behaving any differently than existing.  This seems
to be a
> > > > >> > >> potentially critical scaling point.  Is there a
way to do
> this
> > with
> > > > >> > >> the Spring solution?
> > > > >> > >>
> > > > >> > >> The persistence point is a good one, though I would
classify
> > that
> > > > as
> > > > >> > >> "not implemented" vs "not possible" (not that you
were).
> > > > >> > >>
> > > > >> > >> I'm not married to Camel, I just like the EIP approach
to
> > building
> > > > >> > >> something that is ultimately a messaging system.
 There are
> > known
> > > > >> > >> patterns that solve at least a subset of the problems
Streams
> > is
> > > > >> > >> trying to solve and implementations that can handle
the load
> > and
> > > > I'll
> > > > >> > >> reiterate flexibility == complexity almost always.
> > > > >> > >>
> > > > >> > >> It comes right back to the central question: Do
you want
> > > > flexibility
> > > > >> > >> or simplicity?  It doesn't have to be black and
white either
> I
> > > > don't
> > > > >> > >> think...
> > > > >> > >>
> > > > >> > >> More pointedly: Where should we give up flexibility
for
> > > > simplicity?  I
> > > > >> > >> read that Danny is advocating for the simplicity
at
> > registration to
> > > > >> > >> give up flexibility at registration, but retaining
the inner
> > > > "guts" of
> > > > >> > >> EIP/messaging?  Thoughts?
> > > > >> > >>
> > > > >> > >> On Wed, Oct 30, 2013 at 11:37 AM, Danny Sullivan
<
> > > > dsullivan7@hotmail.com> wrote:
> > > > >> > >> > My argument is not for the IoC pattern as
that can be (and
> > has
> > > > been) implemented alongside Camel. My main argument is that the
> syntax
> > at
> > > > the entry point is not only familiar but much simpler. This wouldn't
> > be a
> > > > very strong argument if the Camel implementation wasn't much more
> > > > complicated but I feel that it is the case. Also, looking toward the
> > > > future, if the server is restarted, in-routes are lost in Camel. The
> > way to
> > > > curb this is to persist the dynamic routes that Camel creates, and
> > then on
> > > > start up pull every one of these routes and recreate a dynamic route
> > for
> > > > each one. Not only is this much easier to implement using the Spring
> > web
> > > > implementation, but it already has been implemented and you can try
> it
> > by
> > > > checking out the webservice branch, registering a subscriber,
> > restarting
> > > > tomcat, and using the same url you had before. This will allow
> > subscribers
> > > > to hang on to their urls once they register. (the same is true for
> > > > publishers: you can post via the same url after restarting tomcat)
> > > > >> > >> >
> > > > >> > >> >> Date: Wed, 30 Oct 2013 11:00:21 -0400
> > > > >> > >> >> Subject: Re: [DISCUSS] Switching Streams
from Camel
> > deployment
> > > > to .war deployment
> > > > >> > >> >> From: jletourneau80@gmail.com
> > > > >> > >> >> To: dev@streams.incubator.apache.org
> > > > >> > >> >>
> > > > >> > >> >> To be fair, while the current implementation
is heavily
> > > > camel-based,
> > > > >> > >> >> all of the interfaces related to Streams
functionality are
> > not.
> > > >  The
> > > > >> > >> >> current model maps to what Matt has outlined
in my
> opinion,
> > > > though
> > > > >> > >> >> packing names etc. probably don't follow
that exact
> pattern.
> > > > >> > >> >>
> > > > >> > >> >> With regards to the complexity and different
components in
> > the
> > > > >> > >> >> registration process, this was a cut at
the abstraction
> > based
> > > > on the
> > > > >> > >> >> assumption that different implementations
may be plugged
> in
> > and
> > > > in
> > > > >> > >> >> fact may live on different processor space
(ie. a polling
> > > > publisher vs
> > > > >> > >> >> a push publisher may be instantiated on
different servers
> > but
> > > > the
> > > > >> > >> >> registration URL is staticly defined).
> > > > >> > >> >>
> > > > >> > >> >> Is the main argument I am seeing for Spring
 the
> > familiarity of
> > > > its
> > > > >> > >> >> IoC pattern implementation and syntax
at the entry point?
> > > > >> > >> >>
> > > > >> > >> >> On Wed, Oct 30, 2013 at 10:53 AM, Danny
Sullivan <
> > > > dsullivan7@hotmail.com> wrote:
> > > > >> > >> >> > Could you clarify whether the same
entry points would
> > exist
> > > > for the camel implementation of the core (implementing the "process"
> > > > method/ using a DynammicRouteBuilder) or would the webservice be the
> > sole
> > > > entry point to Streams and after it enters would it hand it off to
> > Camel?
> > > > And what would be the entry point for the Storm implementation?
> > > > >> > >> >> > -Danny
> > > > >> > >> >> >
> > > > >> > >> >> >> Date: Wed, 30 Oct 2013 10:13:04
-0400
> > > > >> > >> >> >> Subject: Re: [DISCUSS] Switching
Streams from Camel
> > > > deployment to .war deployment
> > > > >> > >> >> >> From: m.ben.franklin@gmail.com
> > > > >> > >> >> >> To: dev@streams.incubator.apache.org
> > > > >> > >> >> >>
> > > > >> > >> >> >> On Tue, Oct 29, 2013 at 2:55
PM, Danny Sullivan <
> > > > dsullivan7@hotmail.com>wrote:
> > > > >> > >> >> >>
> > > > >> > >> >> >> > My case for switching from
OSGi is for simplicity in
> > > > design. To follow the
> > > > >> > >> >> >> > path of an activity through
Streams in the
> webservice,
> > > > there is one main
> > > > >> > >> >> >> > things the developer needs
to understand:
> > > > >> > >> >> >> > The @RequestMapping annotation
specifies the HTTP
> entry
> > > > point
> > > > >> > >> >> >> > There are 4 @RequestMapping
annotations that
> > correspond to
> > > > each of the 4
> > > > >> > >> >> >> > ways a user enters the application:
registering a
> > > > publisher, registering a
> > > > >> > >> >> >> > subscriber, publishing activity,
and getting an
> > activity
> > > > stream. Where
> > > > >> > >> >> >> > these are located in the
source code can be found by
> > > > searching for the
> > > > >> > >> >> >> > paths specified in the documentation
(search for
> > > > "/publisherRegister",
> > > > >> > >> >> >> > "/subscriberRegister", "/publishActivity",
> > "/getActivity"
> > > > which will all
> > > > >> > >> >> >> > lead you to StreamsWebController.java).
From the
> > methods
> > > > that process
> > > > >> > >> >> >> > requests, the flow through
the application is through
> > > > methods which can be
> > > > >> > >> >> >> > understood by most Java
programmers.
> > > > >> > >> >> >> > The flow of activities through
the current trunk
> > branch is
> > > > understood as
> > > > >> > >> >> >> > follows:
> > > > >> > >> >> >> > The string "/publisher/register"
(the entry point to
> > > > register a publisher
> > > > >> > >> >> >> > specified in the documentation)
is the value of the
> > > > >> > >> >> >> > consumer.registrationEndpoint
property defined in
> > > > streams.propertiesThe
> > > > >> > >> >> >> > camelContext.xml specifies
an endpoint with the id
> > > > >> > >> >> >> > consumerRegistrationEndpoint
to have a uri equal to
> the
> > > > propertyThe
> > > > >> > >> >> >> > consumerRegistrationEndpoint
routes from uri
> > > > direct:publisher register with
> > > > >> > >> >> >> > the bean activityRegistrationProcessor
nested in
> > between
> > > > the routeThe
> > > > >> > >> >> >> > streams-eip-applicationContext
contains a bean with
> > the id
> > > > >> > >> >> >> > activityRegistrationProcessor
created for the class
> > > > >> > >> >> >> > ActivityPublisherRegistrationProcessorThe
exchange
> will
> > > > enter the "process"
> > > > >> > >> >> >> > method of the class
> > > > ActivityPublisherRegisitrationProcessor and that this
> > > > >> > >> >> >> > is because this class implements
the "Processor"
> > interface
> > > > provided by
> > > > >> > >> >> >> > CamelThe direct:add-publisher-route
takes the
> exchange
> > > > output from the
> > > > >> > >> >> >> > "process" method and routes
it to the
> > > > activityRegistrationProcessor
> > > > >> > >> >> >> > "register" method. The bean
> > activityRegistrationProcessor
> > > > is defined in the
> > > > >> > >> >> >> > streams-eip-osgi-component-import.xmlThe
output from
> > this
> > > > method is then
> > > > >> > >> >> >> > sent to the "createNewRouteForConsumer"
method of
> > > > activityConsumerRouter.
> > > > >> > >> >> >> > This method creates a new
route for the newly
> > registered
> > > > publisher using
> > > > >> > >> >> >> > the private static final
class
> > DynamicConsumerRouteBuilder
> > > > which is
> > > > >> > >> >> >> > required to extend RouteBuilder
which is provided by
> > > > Camel. This
> > > > >> > >> >> >> > DynamicConsumerRouteBuilder
contains several methods:
> > > > >> > >> >> >> > getConsumerReceiveMethod()
(which corresponds to
> @Value
> > > > >> > >> >> >> > ${consumer.receiveMethod}
which corresponds to
> > "receive"),
> > > > >> > >> >> >> > getConsumerSplitMethod()
(@Value
> > > > ${consumer.splitMethod},"split"), and
> > > > >> > >> >> >> > getConsumerActivityQUri()
(@Value
> > ${consumerActivityQUri},
> > > > >> > >> >> >> > "direct:activityQ"). This
is different than the
> > > > camelContext.xml in that
> > > > >> > >> >> >> > the route is being created
programmatically. What
> this
> > is
> > > > doing is routing
> > > > >> > >> >> >> > input from the inroute url
(which Camel does
> > automatically
> > > > through the
> > > > >> > >> >> >> > configure method which is
required to be overridden),
> > to
> > > > the "receive"
> > > > >> > >> >> >> > method of ActivityConsumer,
then to the "split"
> method
> > of
> > > > ActivityConsumer,
> > > > >> > >> >> >> > and then to the "direct:activityQ"
which if you look
> > back
> > > > in the
> > > > >> > >> >> >> > camelContext.xml routes
to the
> > "activemq:queue:activities"
> > > > which then
> > > > >> > >> >> >> > routes to "receiveExchange"
> > > > >> > >> >> >> > This is the process to register
a publisher. The
> > process
> > > > for registering a
> > > > >> > >> >> >> > subscriber is relatively
the same though it involves
> > > > separate classes with
> > > > >> > >> >> >> > their own private static
final RouteBuilder class.
> > From my
> > > > perspective, the
> > > > >> > >> >> >> > two most difficult things
with setting this project
> up
> > > > were understanding
> > > > >> > >> >> >> > that the "process" method
of the class that
> implements
> > > > "Processor" is the
> > > > >> > >> >> >> > entry point and the DynamicConsumerRouteBuilder
> creates
> > > > the second entry
> > > > >> > >> >> >> > point (The 5th and last
points). This made the
> project
> > > > very, VERY hard to
> > > > >> > >> >> >> > understand.
> > > > >> > >> >> >> > In addition to simplicity
of design, the mvn clean
> > install
> > > > of the web
> > > > >> > >> >> >> > service project is much
faster and small scale
> activity
> > > > publishing is also
> > > > >> > >> >> >> > faster (see my email about
load testing). These are
> > minor
> > > > points though as
> > > > >> > >> >> >> > compilation has no effect
on deployment. OSGi does
> add
> > the
> > > > benefit of
> > > > >> > >> >> >> > modularized programming
which is valuable, though I
> > think
> > > > the added
> > > > >> > >> >> >> > complexity of Camel merits
moving the project away
> from
> > > > this paradigm.
> > > > >> > >> >> >> >
> > > > >> > >> >> >>
> > > > >> > >> >> >> I agree that the project is pretty
difficult to
> > understand
> > > > ATM.  I think
> > > > >> > >> >> >> what we need to do is think about
what the
> > responsibilities
> > > > of the code are
> > > > >> > >> >> >> and allow for different implementations
that are not so
> > > > tightly coupled as
> > > > >> > >> >> >> they are now.  For instance,
having worked with Storm
> to
> > > > ingest millions of
> > > > >> > >> >> >> activities a day, I personally
would like to see
> streams
> > be
> > > > responsible for
> > > > >> > >> >> >> defining an over-arching orchestration
model that can
> be
> > > > implemented within
> > > > >> > >> >> >> a single war or on top of a distributed
system.  This
> > would
> > > > look something
> > > > >> > >> >> >> like the follows:
> > > > >> > >> >> >>
> > > > >> > >> >> >> |__ Streams-Core  (Base classes,
interfaces, utilities,
> > > > extensions, etc)
> > > > >> > >> >> >> |
> > > > >> > >> >> >> |__ Streams-Storm (Storm implementation
of the core)
> > > > >> > >> >> >> |
> > > > >> > >> >> >> |__ Streams-Camel (Camel implementation
of the core)
> > > > >> > >> >> >> |
> > > > >> > >> >> >> |__ Streams-WS (Web service implementation)
> > > > >> > >> >> >>
> > > > >> > >> >> >>
> > > > >> > >> >> >> > Danny
> > > > >> > >> >> >> >
> > > > >> > >> >> >> >
> > > > >> > >> >> >> > > From: dsullivan7@hotmail.com
> > > > >> > >> >> >> > > To: dev@streams.incubator.apache.org
> > > > >> > >> >> >> > > Subject: [DISCUSS]
Switching Streams from Camel
> > > > deployment to .war
> > > > >> > >> >> >> > deployment
> > > > >> > >> >> >> > > Date: Tue, 29 Oct 2013
11:07:39 -0400
> > > > >> > >> >> >> > >
> > > > >> > >> >> >> > > The discussion thread
for switching Streams from
> > > > Camel/osgi/Servicemix
> > > > >> > >> >> >> > to a single .war deployment
> > > > >> > >> >> >> >
> > > > >> > >> >> >> >
> > > > >> > >> >> >
> > > > >> > >> >
> > > > >> > >
> > > > >>
> > > > >
> > > >
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message