activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gaurav Sharma <gaurav.cs.sha...@gmail.com>
Subject Re: Concerns about ActiveMQ implementation over the WAN
Date Thu, 27 Sep 2012 09:56:57 GMT
So, there are many ways to skin this but let's consider one of the
'simple' solutions given your constraints primarily due to the
remoteness.

Given that:
1. the system as described seems to be doing scatter-gather with
bi-directional data-transfer - master pushing jobs to workers, workers
pushing results to master.
2. it appears that your 'clients / workers' have some locale-specific
information that the central-server / 'master' does not have for
processing the 'jobs'. Otherwise, it makes no sense to pay the cost of
shipping all this data and processing to remote workers.
3. a key consideration is heterogeneity of the worker platforms.

There is other missing information:
1. How fault tolerant is the master? Throughput is rather easier to accomplish.
2. How fault tolerant are the workers? If output of a job is lost, can
it be regenerated?
3. Is there an SLA around end-of-job-execution to shipping-of-results to master?
4. What kind of WAN links are available between the workers and the
master? What about the local storage on the worker cluster?
5. It appears that your 'network' is running on the public internet.
VPN can help; if not, REST over HTTPS is your friend.
6. Can the master not let workers 'pull' jobs instead of 'pushing' them?

A v1 solution could look something like:
1. A RESTful endpoint on the Master to accept job-results as HTTP POST's
2. A RESTful endpoint for every worker-cluster to accept jobs for execution
3. 1 AMQ broker instance in the master, 1 each per worker-cluster;
brokers support durability/persistent messages with TTL's
4. master-broker instance can have 2 queues: job-dispatcher,
job-results-gatherer
5. worker-broker instances can each have 2 queues: job-receiver,
job-results-dispatcher
6. The REST endpoints are just facades to support transport and
provide platform independence
7. AMQ queues provide resilience even when your WAN links are offline.
Also, much better to collect/gather results and dispatch to master as
a batch rather than trickle dispatching
8. Do not put xml files in the queues. Stream files to shared
storage/disk and put metadata/file-locations in queues.
9. Dispatcher and Receiver threads should handle the data streaming,
compression, queue get/put operations, etc.
10. With this design, you can independently tune shipping-frequencies
of job-delegations (master->workers) and that of job-results
(workers->master).
11. Replay of failed jobs should be doable given you have two queues
in every worker-broker cluster
12. AMQ has language-specific client-libraries, so, your heterogeneous
worker-brokers can talk their respective languages (java, cpp, .net,
scripting, etc).
13. AMQ supports many broker topologies. If you have some kind of a db
available, you can use the master-slave with db topology on every
broker.

High-level flow:
1. A Master-job-creator thread pushes worker-job message to the
job-dispatcher-queue
2. Master-job-dispatcher thread periodically polls the
job-dispatcher-queue for pending jobs to dispatch to workers
3. It finds a pending job, does an HTTP POST to the appropriate worker
cluster's job-acceptor endpoint. If successfully acked (2xx, removes
message from job-dispatcher-queue).
4. Worker's job-acceptor endpoint pushes the job as a message to its
local cluster broker's job-receiver-queue.
5. Worker's job-processor thread polls the job-receiver-queue for new
job messages, finds one to process and does its thing.
6. After successful processing, the job-processor thread removes the
message from the job-receiver-queue and pushes job-result-metadata to
the job-results-dispatcher queue
7. Worker's job-results dispatcher thread polls the
job-results-dispatcher queue for messages, finds one, does an HTTP
POST to the master's job-results acceptor endpoint.
8. If your business rules allow, you could batch the results and tune
batch-size vs shipping-frequency.
9. Master's job-results acceptor endpoint pushes job-results as a
message to job-results-gatherer queue from where the
results-processing threads can take over.
10. Polling queues is one option with the other option as using
MessageListener's to let the broker do event-signaling via
onMessage().
11. Now, given that you have durable destinations in all locations,
you can shut down any/all machines (master and workers) that run the
processing logic, switch off networks, survive network outages, etc,
as long as the system doesn't fall behind too much and then it becomes
a capacity question.
12. I made an assumption of queue-push/pull processors being threads;
they could very well be processes depending upon your system.
13. I don't know how your system might evolve and whether the xml
payload can grow but 5-10K is not very large for a textmessage if you
want to just store it in the destination directly, that will also work
at least for the moment.

Hope this helps!

-gaurav


On Wed, Sep 26, 2012 at 10:28 PM, kmansari <kamran_ansari@yahoo.com> wrote:
> My sincere apology if this has been already addressed elsewhere.
>
> I am working on an architectural initiative, the scope of which is pretty
> broad compared to what I have done in the past. I thought it prudent to run
> it by the experts and others who may be more experienced than me in this
> domain.
>
> The system in question is a distributed one, spanning multiple continents.
> It involves clusters of clients (about 100 or so), running across a mix of
> OS/platforms, in each continent. For sake of simplicity, let’s refer to each
> cluster as a “Job Site” (the number of these “Job Sites” is expected to grow
> over time). Each client in a given cluster performs a set of tasks,
> requested by a central server. These are long running tasks that could take
> from a few hours to a couple of days. Once a client is done with its job, it
> has to communicate the results of the job (an XML file, 5K to 10K in size)
> to the central server. The client needs a reliable and guaranteed way to
> convey the results back to the central server, because as soon as it is done
> with its tasks, it has to reboot and get ready for the next job that might
> be coming its way. The central server scales well and has the capacity to
> handle potential spikes in the incoming traffic.
>
> I concluded that it made sense to implement the communication layer using a
> “Message Bus”. ActiveMQ, in particular, piqued my interest primarily because
> it is cross platform, robust, reliable, and scales well. My proposal
> articulated that the clients will communicate with the ActiveMQ broker (most
> likely in a client-server topology), directly, without an intermediate
> web-services interface sitting in front of the message bus.
>
> Some stakeholders in my team have expressed the following concerns:
>
> 1.      *Performance Concern:* A Message Bus doesn’t perform well on the WAN.
> After doing a search on the forum, it seems like a “Network of Brokers”
> topology would be able to address the performance related concern. Are there
> any more arguments besides the “Network of Brokers” to alleviate the
> performance concerns?
>
> 2.      *Security Concern:* This is the bigger of the two concerns raised. The
> lack of a web-services interface for the clients to interact with the
> message bus “exposes the message bus to the internet, thus making it
> vulnerable to security threats”.  Among the solutions proposed to mitigate
> the risk:
> a.      Use a web service instead of message bus.
> b.      Implement a web-service layer in front of the message bus. In my mind,
> that defeats the whole purpose of using a message bus in the first place.
> One might as well throw away the message bus and completely replace it with
> a REST/SOAP web service interface.
>
> Do the aforementioned concerns really have any merit? How are others
> implementing ActiveMQ based solutions on the WAN? Any other issues that I
> should be aware of or concerned about? Your thoughts are greatly
> appreciated.
>
> Thanks much!
>
> -Kamran
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Concerns-about-ActiveMQ-implementation-over-the-WAN-tp4657072.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Mime
View raw message