activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "James A. Robinson" <j...@highwire.org>
Subject Re: Network of Brokers recommendations
Date Mon, 27 Apr 2015 17:25:46 GMT
Hi,

Thanks for the reply.  My earlier distinction was sloppy, I ought to have
written something like 'partially connected mesh' vs. a complete graph.   I
was trying to distinguish between a topology where nodes are only connected
to immediate neighbors, and therefore may always be required to travel
across multiple brokers to get to a particular destination.

In this particular setup all the brokers will have the same capabilities
and will be connected to the same network, so I think I'll go ahead with
these scheme.  I appreciate the feedback!

Jim

On Sun, Apr 26, 2015 at 9:10 AM Tim Bain <tbain@alumni.duke.edu> wrote:

> Your understanding of this is good and you've got the right concepts here.
> The only two reasons I can think of that you shouldn't use a mesh (which I
> think is the same as a complete graph, though you drew a distinction
> between the two in your email so maybe there's a difference I'm not
> understanding) is if you have non-homogeneous network links where the most
> efficient routing between two brokers isn't necessarily the path "directly"
> (from a network-of-brokers standpoint, not from a network standpoint)
> between the two, or if you don't want to have to specify all of the other
> clusters so you'd use a hub-and-spoke topology so you only have to specify
> the hub node in your config.
>
> On Fri, Mar 13, 2015 at 12:21 AM, James A. Robinson <
> jim.robinson@gmail.com>
> wrote:
>
> > Hi folks,
> >
> > I joined this list because we're starting to look into setting up
> > ActiveMQ for our developers.  My preference at this time is to set
> > up a highly reliable system that doesn't have single points of
> > failure.
> >
> > Due to the concern about single points of failure I didn't want to
> > introduce a shared persistence layer.  That's why I've been looking
> > at replicated LevelDB.
> >
> > After spending some time reading up on ActiveMQ my first inclination
> > has been to propose setting up 3-node replicated LevelDB clusters,
> > and to build a network of brokers out of sets of those 3-node
> > clusters.
> >
> > E.g., start with two clusters, and then keep adding on as we see
> > the need in terms of capacity.
> >
> > [amq-1a, amq-1b, amq-1c]
> > [amq-2a, amq-2b, amq-2c]
> > ...
> > [amq-9a, amq-9b, amq-9c]
> > ...
> >
> > I'm not sure what the best option is w/re to the network of broker
> > topology.  I could see mesh working, but my first instinct would
> > be to set up a complete graph, and have each broker cluster configured
> > with a uni-directional network connector to every other cluster
> > using the masterslave scheme.
> >
> > I am wondering if this is a naive approach, and if so why it wouldn't
> > be a good idea.
> >
> > If it is reasonable, I'm still trying to fully understand how one
> > deals with forwarding messages from brokers w/o consumers to those
> > brokers w/ consumers.  The page
> >
> > http://activemq.apache.org/networks-of-brokers.html
> >
> > discusses this topic and mentions rebalanceClusterClients, and it
> > isn't clear to me how that option will help if the clusters and
> > clients are fairly static:
> >
> >   rebalanceClusterClients:
> >     if true, connected clients will be asked to rebalance across a
> >     cluster of brokers when a new broker joins the network of brokers
> >     (note: priorityBackup=true can override)
> >
> > Reading the list of options available for transport connectors, it
> > seems reasonable to configure the transport connector with:
> >
> > updateClusterClients="true"
> > rebalanceClusterClients="true"
> > updateClusterClientsOnRemove="true"
> >
> > But if one has mostly static clients and we aren't frequently
> > bringing broker clusters up and down, I would assume there would
> > still be a problem of having messages on a broker that doesn't have
> > consumers attached.  If I'm wrong I'd like to understand what I'm
> > missing.
> >
> > The network-of-brokers page, after mentioning rebalanceClusterClients
> > as one solution, continues the discussion and states that:
> >
> >   Another, is to allow replay of messages back to the origin as
> >   there is no local consumer on that broker.
> >
> >   There is a destination policy that allows this behavior for queues
> >   by configuring a conditionalNetworkBridgeFilterFactory with
> >   replayWhenNoConsumers=true. The conditionalNetworkBridgeFilterFactory
> >   provides an optional replayDelay based on the broker-in time.
> >
> >   N.B.: When using replayWhenNoConsumers=true for versions < 5.9,
> >   it is necessary to also disable the cursors duplicate detection
> >   using enableAudit=false as the cursor could mark the replayed
> >   messages as duplicates (depending on the time window between
> >   playing and replaying these messages over the network bridge).
> >   The problem is fully explained in this blog post.
> >
> > And I see in that ticket AMQ-4607 says this about the network
> > connector options:
> >
> >   networkTTL sets both consumerTTL and messageTTL a value of -1
> > denotes infinite hops.
> >   In a mesh, use consumerTTL=1, messageTTL=-1 to allow consumers to
> > bounce around the mesh and messages to follow demand.
> >   In a linear topology use networkTTL == length of network (as before)
> >
> > So would it be reasonable to set up a complete graph, with each
> > 3-node cluster configured with a uni-directional network connector
> > to every other 3-node cluster using masterslave, and to set
> >
> >   consumerTTL="1"
> >   messageTTL="-1"
> >   decreaseNetworkConsumerPriority="true"
> >
> > and a networkBridgeFilterFactory/conditionalNetworkBridgeFilterFactory
> > with
> >
> >   replayWhenNoConsumers="true"
> >
> > I'd appreciate any input from the folks here, including any pointers
> > on any more material I should read.
> >
> > Jim
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message