activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From bacevt <bac...@gmail.com>
Subject sender side manual load balancing
Date Wed, 16 Jun 2010 14:03:48 GMT

Hi,

I am trying to find the best scalability setup regarding my JMS scenario,
and I would like to ask you to help me to do that. The scenario is very
simple – I have a set of producers, which send thousands of persistent
messages on a queue for a several seconds, as a consequence of some UI
activity. I also have a set of consumers, responsible for message
processing. The order of message processing is not important – each message
is self contained, and its processing is completely independent of the
other’s messages processing. The processing is a very complex, and typically
it takes one or several minutes to complete the message processing.

In order to improve my application scalability, I would like to use several
independent brokers and delegate the decision to the sender application
which broker would to send a message to.
I don’t think to use a single broker, /despite of the fact that this is a
highly recommended solution/ because I don’t want to have a single point of
failure. Network of brokers is a useful option for me, but I am afraid that
this will introduce addition network overhead. This “sender side manual load
balancing” combines the best of the both worlds – no single point of failure
and no cluster communication overhead. 

Probably this scenario is very similar to the jedi-transport request: 
https://issues.apache.org/activemq/browse/AMQ-816
However, jedi-transport is still unavailable, so it is not a real option. 

In order to delegate the decision to choose a broker to which to send a
message, the sender application should be able to know the list of all
available brokers. One possible solution is to use a hard coded list,
however a more advanced solution will be able to detect dynamically
available brokers. The idea is to implement discovery listener on the client
side and to listen for discovery events and to maintain a set of available
brokers. The sender application will be able to send a message using round
robin load balancing between brokers.

On the other hand I need to use broker’s discovery mechanism, but with some
modification – I expect brokers to expose themselves, but not to establish a
network of brokers between them. I checked the code, and regarding my
findings the only possible way to do that is to setup each broker in
different group and to share one and the same multicast address. For
example:

Broker1 -> discoveryUri="multicast://default?group=broker1"
Broker2 -> discoveryUri="multicast://default?group=broker2"
…
BrokerN -> discoveryUri="multicast://default?group=brokerN"

This setup will challenge all brokers to advertise themselves and at the
same time each broker belongs to its own group. I don’t expect additional
network traffic due to clustering, except the packets sent by discovery
agents. This will provide a solution for membership detection to the sender
application – it simply listens on the same multicast address and analyses
the incoming notifications. For example:

received: broker1.ActiveMQ-4.alive.%localhost%tcp://mymachine:61616
received: broker2.ActiveMQ-4.alive.%localhost%tcp://mymachine:61618
received: broker1.ActiveMQ-4.alive.%localhost%tcp://mymachine:61616
received: broker1.ActiveMQ-4.dead.%localhost%tcp://mymachine:61616
received: broker2.ActiveMQ-4.alive.%localhost%tcp://mymachine:61618
received: broker2.ActiveMQ-4.dead.%localhost%tcp://mymachine:61618

Probably this is very tricky and risky solution, since the client side
application should analyze a data packets, used only internally. I need to
re-implement the logic of MulticastDiscoveryListener with the main aim to
implement DiscoveryListener. The implementation of this listener will
receive notification about service start/stop and will maintain a set of the
available subscribers.
The sender application will be able to discover dynamically all available
brokers and to use some load balancing strategy to utilize them.

Do you think that my approach is siutable?
Do you see any serious downfalls in my suggestion? 
Any help will be highly appreciated.


Greetings
-- 
View this message in context: http://old.nabble.com/sender-side-manual-load-balancing-tp28903218p28903218.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.


Mime
View raw message