activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gaurav Sharma <gaurav.cs.sha...@gmail.com>
Subject Re: Concerns about ActiveMQ implementation over the WAN
Date Fri, 28 Sep 2012 08:52:38 GMT
So, there are always trade-offs to be made when considering options.
Sure, you could go a step further and just use oracle streams across
continents and somehow make it work without messaging - is it
desirable? Most would say not.

You could also try to create a distributed broker grid like you have
in mind (fault-tolerance and message delivery guarantees will
certainly be key issues to address) but with a little additional work
of creating the REST endpoints, you get so many more options and
flexibility. It shouldn't take more than 30minutes to get those two
endpoints running if you used something like Spring/Jersey and
deployed on Jetty and they will serve as nice forwarding proxies. You
could add rate-limiting, role-based access-controls, retries,
timeouts, ping-url's, etc to these with little effort in the future.
This will let you insulate your brokers from each other, eg.:
- you could upgrade worker sites to a newer version of AMQ
independently of each other without worrying about having to worry
about your global network updating in lock-step
- you won't even have to run AMQ on all your worker sites. they just
talk REST/HTTP with the master. some worker sites might want to run
hornetq or msmq/.net
- and going a step further, it gets you complete broker-independence:
switch out the broker implementation everywhere if you wanted to

Also, regarding "knowing about each other", IIRC per the original
description, the master dispatches jobs to the workers and the workers
dispatch job-results back to the master. So, that implies
bi-directional communication: the master does know all the worker
uri's and all the workers know the master uri. I never said that the
worker endpoints need to know about other worker endpoints, just the
master uri.

If you are worried about extra hardware, you could just fire up the
jetty instance that AMQ comes with and re-purpose it to deploy your
REST service on the same node that runs the broker (master broker, all
worker brokers). Just drop the service war in $amq_home/webapps and
adjust configs in conf (jetty.xml, realm properties, import jetty.xml
in activemq.xml, etc). It is not the most optimal in terms of resource
utilization but it will work. Then just add nagios or whatever
monitoring checks you have for that one node's processes. This way you
can still achieve the same architecture and reap its benefits.

Again, this is just one option; if you want to create a distributed
broker-grid directly wiring together forwarding-brokers, would love to
hear about the experience once you have it productionized.


On Thu, Sep 27, 2012 at 10:36 PM, kmansari <kamran_ansari@yahoo.com> wrote:
> Gaurav,
> Thanks you so much for your detailed response. To answer some of your
> questions briefly:
>
> The master is very resilient, and fault tolerance is not an issue. As for
> the workers, it is possible to generate the output but not without a
> prohibitively large time penalty. There is no end-of-job-execution SLA. And
> you’re right that the network is running on the public internet.
>
> That said, I like your v1 solution, but I am somehow not convinced that it
> cannot be designed without the RESTful endpoints. Some of the reasons I
> would like to avoid those endpoints:
>
> •       It adds to infrastructure and maintenance costs
> •       Requires additional development and maintenance of the RESTful endpoints
> •       The worker-cluster and the master have to “know” about each other, by
> having to know each other’s endpoints
>
> Keeping in mind those reservations and making an assumption for the purpose
> of this discussion that supporting heterogeneity is not an issue or somehow
> could be address using other means, I wonder if the worker-cluster and the
> master could talk to each other using OpenWire over TCP transport
> (tcp://<broker>:61616), without having to rely on the RESTful endpoints. The
> AMQ instances at the job-sites would act as the forwarding-brokers to the
> master instance. All messages to job-sites could be filtered on a unique
> “site-id”. Now, take this a step further: imagine there are three job-sites
> in China – they can share a single instance of AMQ forwarding broker (could
> be failover-configured multiple instances), instead of having one forwarding
> broker per site. Is that even feasible in your experience? Impractical? I
> don't know what I am talking about?
>
> Thanks again for your very well articulated and detailed response. Much
> appreciated!
>
> Best,
> -Kamran
>
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Concerns-about-ActiveMQ-implementation-over-the-WAN-tp4657072p4657137.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Mime
View raw message