activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tim Bain <>
Subject Re: Zero consumers on some brokers after reboot
Date Sun, 26 Jun 2016 20:52:44 GMT
To unsubscribe, email, not the list
On Jun 26, 2016 7:44 AM, "vas u" <> wrote:

On Jun 24, 2016 10:16 AM, "FrankS" <> wrote:

> Hello,
> I am working on a project that uses a network of ActiveMQ brokers.  We use
> a
> fixed set of network connectors.  The test team has been performing outage
> tests by shutting down services, killing services, and rebooting servers.
> After the outage test, they give the system a moment to recover, then they
> check the consumer counts for the queues and topics.  They found that
> sometimes, the consumer counts on a few queues are inconsistent between
> brokers.  The client has connected to the broker without error (a local
> connection), and that broker shows a consumer.  But the broker with a
> producer connection has a consumer count of 0.  In this situation, the
> producer is still trying to send messages, but they never get to the
> consumer.
> Does anyone have advice on how to debug this?  We've turned on debug, but
> its really noisy and we are not sure if there are specific messages or
> loggers that we should be looking for.
> We have seen some errors on other brokers when a rebooted system comes
> and tries to connect again (this was when the broker wasn't shut down
> nicely).  They other brokers seem to think they still have a connection to
> that broker (no TCP SYN sent from rebooted broker I assume?).  I'm
> wondering, could it be that there is some sort of race condition when the
> original connection is cleaned up while a new one is created, and perhaps
> the new advertisement for consumer interest is cleaned up with it?
> A few things we have noticed that seem to contribute to the problem:
> * The AMQ service was not setup correctly to stop nicely on reboot, so it
> was likely just killed.  When this was corrected, we had difficulty
> reproducing the problem.
> * It seems limited to queues the consumer creates manually using an
> ActiveMQConnectionFactory.  The camel queues don't seem affected (sorry,
> not
> too familiar with camel yet).
> The ActiveMQConnectionFactory is created like this:
> Then a connection pool is created with it ("rawFactory" is the factory
> created above):
> This is a broker config from one of the servers.  They are all the same,
> except for the tweak to the networkConnectors list.
> Any hints or tips would be greatly appreciated!
> Thank you,
> Frank
> --
> View this message in context:
> Sent from the ActiveMQ - User mailing list archive at

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message