activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matt Pavlovich (JIRA)" <>
Subject [jira] [Commented] (AMQ-2875) Create a <treeConnector> for ActiveMQ to provide better support for hierarchical broker topologies
Date Wed, 06 Mar 2013 15:24:12 GMT


Matt Pavlovich commented on AMQ-2875:

Hi Adrian-

I definitely agree that there is room for improvement on the networkConnector/network of brokers
front to make it more usable, specifically in the large-scale scenarios you describe. 

A couple of comments:

1. Any improvement or modification to networkConnector-type configuration would be greatly
enhanced by having the configuration be dynamically updatable and persisted. (ie.. save JMX/REST
config changes to the activemq.xml file)

2. What do you think about supporting the "host" portion of the JMS url for routing? It seems
to me that what you are describing might be achieved by supporting that, instead of tracking
advisories up and down nodes. Having the ability to support routing a message to "kiosk46"
when specifying the destination as "queue://kiosk46/MY_QUEUE" and client sending message to
"brokerA" (HQ) might be easier to implement and manage than a number of network/tree connectors.
This would also make migrations from commercial platforms that currently support this, such
as MQ series, more straightforward. Thoughts?

In my personal experience from implementing ActiveMQ at a global organization with 1000's
of nodes is that the number of queues can become problematic in a straight-forward network-of-brokers
model. Also, if all the endpoints have unique queues, it pushes the complexity to the application
developers which creates a negative developer experience (imho).

[Preferred-- Apps deployed to kiosk46 know to connect to queues and don't need to know anything
about its current host]
Host: Kiosk46
  - queue:///ORDERS_Q
  - queue:///TICKETS_Q
  - queue:///INVENTORY_Q
Host: Kiosk48
  - queue:///ORDERS_Q
  - queue:///TICKETS_Q
  - queue:///INVENTORY_Q

[Versus-- Apps need to consider their location when creating queue names]
Host: Kiosk46
  - queue:///K46_ORDERS_Q
  - queue:///K46_TICKETS_Q
  - queue:///K46_INVENTORY_Q

host: Kiosk48
  - queue:///K48_ORDERS_Q
  - queue:///K48_TICKETS_Q
  - queue:///K48_INVENTORY_Q

Matt Pavlovich
> Create a <treeConnector> for ActiveMQ to provide better support for hierarchical
broker topologies
> --------------------------------------------------------------------------------------------------
>                 Key: AMQ-2875
>                 URL:
>             Project: ActiveMQ
>          Issue Type: New Feature
>            Reporter: Adrian Trenaman
> The ActiveMQ network connector is excellent at facilitating self-organizing networks
of ActiveMQ brokers with dynamically discovered routing, and can be used in situations where
network availability is sporadic and not guaranteed. Network connectors can be used along
with master-slave pairs to create a distributed 'messaging fabric'. The network connector
can also accommodate the creation of hierarchical networks, however, I believe there is a
good argument for treating tree topologies as a special case in their own right. In this note,
I'm going to describe why trees should be treated as a special case, and describe what a treeConnector
might look like.
> Hierarchical (or 'tree') networks are a special case of the abstract notion of a network;
they have numerous applications in the real world in cross-geography, wide-area deployments.
Consider a retail organization with 1,000's of stores, who wish to send and receive information
to and from head-quarters (HQ) and the stores: you can envisage a hierarchy of ActiveMQ brokers:
one for HQ, one for each of the regions (e.g. Ireland, UK, France, Germany), and then a broker
in each store/outlet in each of the regions. When HQ wants to send a message to a store, it
should be able to write a message to the store's queue and have it dynamically routed to the
store via the regional broker. Likewise, if a store wants to send a message to HQ, it should
be able to write to a local queue in the store and have the message dynamically routed to
> Right now in ActiveMQ, there are a number of issues with using a network connector to
achieve a tree topology: 
> * Spillage of consumer advisories. When a consumer connects to a store, an advisory message
is generated and sent to all brokers within the networkTTL range. These advisories don't just
travel up the tree to the top (HQ): they can also spill back down the tree to peers who are
simply not interested. We have seen this in a network with 1,000 brokers connected to a regional
broker, and a networkTTL of two: the advisory gets sent to the regional broker, HQ, *and*
the 999 peers of the broker. 
> * Resource wastage (as a consequence of spillage). When a consumer advisory from a store
broker (say, broker 'A') reaches another broker within the networkTTL (say, broker 'B'), then
'B' creates a subscription for the destination (to do forwarding) *and* can result in the
allocation of a thread for this destination. So, it's possible that Broker B will allocate
a thread for the destination 'BrokerA.Incoming', despite the fact that it will never receive
or send to this queue. If there are only a small number of peers at this level then this is
not a problem; however, if there are many peers (as in the case of 1000 stores per region)
then this will be noticeable.
> * Sensitivity to networkTTL. In order to minimize spillage and resource wastage, you
need to se the networkTTL on each broker to the distance between the broker and the root -
in this example, 2. However, a later reconfiguration of the network could result in this setting
being too low (which means messages won't go the distance) or too high (which means you get
inadvertent spillage)
> These issues can be addressed if we create a new tree connector for ActiveMQ. 
> 1) Brokers would identify themselves as 'leaf', 'branch' or 'root' nodes. There should
only ever be one root. If a broker has not been identified as either 'leaf' or 'branch' then
it should consider itself to be the root.
> 2) Tree connectors would be configured using a 'many to one' approach. Leaf brokers would
configure a tree connector to their branch. Branch brokers would configure a tree connector
to the root. The root shouldn't need to configure anything, except of course a transport listener.

> So, a branch broker for the 'UK' region might have the following  (to connect to headquarters):

> 	<treeConnector nodeType="branch" uri=""tcp://"/>
> And, a leaf broker at a store or outlet in the UK might have the following (to connect
to the 'UK' region'): 
> 	<treeConnector nodeType="leaf" uri=""tcp://"/>
> 3) If a consumer connects to a leaf node, then the consumer advisory travels up the tree,
all the way to root: the concept of networkTTL is ignored. If this advisory travels through
a 'branch' then the branch delegates the advisory upwards. Branches do not pass advisory messages
from children to their other child nodes.
> 4) If a consumer connects to a branch node, then the consumer advisory travels up the
tree. Additionally, the branch broker will send the advisory downwards to its children.
> 5) If a consumer connects to the root, then the consumer advisory is filtered downwards
to all children.
> In this way, we reduce spillage of advisory messages, and ensure that trees can self
organize without too much intervention or worrying about the network time-to-live.
> Thoughts?

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see:

View raw message