activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Bastille" <>
Subject High message rate research
Date Wed, 06 Feb 2008 18:08:56 GMT

I'm investigating using ActiveMQ for in an application with a very high
transaction rate.  This is actually a follow up to a thread that Marc
started up a few months ago,
and all of that discussion fully applies here.  The quick summary is:
- Large deployment, expect at least 100 hosts.
- ~200k messages/sec to multiple destinations
- Main concerns are scalability and high-availability, without persistence.
- We do not need guaranteed delivery, 99.9999% is good enough.

>From the research I've done, the only solution that seems viable is
configuring a network of embedded brokers.  Hub/spoke doesn't scale the way
we need it to, and a regular network of brokers seems to have too much risk
of hot spots.  That last statement I'm sure is debatable, as technically
there is nothing different from what I am planning and what a network of
brokers is, however factors such as configuration management make embedding
the broker easier to conceptualize and manage than running many standalone

Assuming a network of embedded brokers, the first part of the topology is
expected to be 50 producer processes, each with an embedded broker, with
each broker connected to each of 20 consumer processes, and each consumer
subscribing to the same distributed queue.  That's 1,000 network
connections, but distributed across 70 hosts, with each broker really having
no more than 50 connections it needs to worry about.

While trying to test this would actually work with ActiveMQ, I found a
couple of problems, each with a work around.  These were tested with both
5.0.0 and 5.1-SNAPSHOT Java API.

1) synchronizing on a BrokerService while calling
ConnectionFactory.createConnection().start() will cause a deadlock.  I
wouldn't say this is a bug, just notable unexpected behavior.

2) Calling Connection.start() after Broker.start() has already been called
leaves a connection with a default broker name.  This probably takes a bit
longer to explain.

So in my testing of the above configuration, I am expecting a dynamic set of
connections to the 20 consumers.  The consumer complex may need to
grow, or get migrated to a different set of hosts, so I'm going to need to
add and remove broker links at runtime.  The main design is that each broker
has a listen port, used or not, and connects to each broker that has a
locally attached consumer that it is interested in.  To support this, I am
adding links after the broker has been started, then calling
connection.start(), and when I have multiple producers connect to a single
consumer, the first connection works fine, but the second connection throws

javax.jms.InvalidClientIDException: Broker: localhost-61616 - Client:
NC_localhost_outbound already connected from /

Unfortunately the stacktrace isn't helpful because this exception is being
thrown by the consumer, and the bug is actually in the producer.
NC_localhost_outbound should have actually been named
NC_localhost-61618_outbound, but when
DemandForwardingBridgeSupport.startRemoteBridge() calls
configuration.getBrokerName(), it returns the default broker name..

Broker.start() calls connection.setBrokerName before calling
connection.start(), but I am adding connections after Broker.start() has
been called, I can get around this by calling connection.setBrokerNamedirectly.

-- End of 2

So far I've only tested 1->many and many->1, and they both seem to work now,
my next test will be many->many, and I'll let you know how that goes, but
this is really just one part of a larger problem.

>From the example above, we can refer to the 50 producers as Complex A, and
the 20 consumers and Complex B.  There is also a complex C of about 20
processes, and a complex D of about 20 processes.  C is a consumer of B, and
D is a consumer of both A and C.

As for message rates:
A -> B: ~200k/sec
A -> D: ~200k/sec
B -> C: ~200k/sec
C -> D: 30M in one burst every hour - can run slower if necessary.

My real question is, given what I've described about this setup so far, will
it work?  Should I expect to run into any circular or inefficient routing
problems? Any other problems?

The first step in this will actually be a bridge between our current system
and this JMS based solution, leveraging the c++ api.  I've read through the
thread from Hellweek (  Were the
problems he found confined to c++ working with c#, or may there also be
problems using c++ with Java?

Thanks in advance,

Sean Bastille

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message