activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tim Bain <tb...@alumni.duke.edu>
Subject Re: Zero consumers on some brokers after reboot
Date Sun, 26 Jun 2016 20:52:44 GMT
To unsubscribe, email users-unsubscribe@activemq.apache.org, not the list
itself.
On Jun 26, 2016 7:44 AM, "vas u" <mailjsr@gmail.com> wrote:

Unsubscribe
On Jun 24, 2016 10:16 AM, "FrankS" <doheth+activemq@gmail.com> wrote:

> Hello,
>
> I am working on a project that uses a network of ActiveMQ brokers.  We use
> a
> fixed set of network connectors.  The test team has been performing outage
> tests by shutting down services, killing services, and rebooting servers.
>
> After the outage test, they give the system a moment to recover, then they
> check the consumer counts for the queues and topics.  They found that
> sometimes, the consumer counts on a few queues are inconsistent between
> brokers.  The client has connected to the broker without error (a local
> connection), and that broker shows a consumer.  But the broker with a
> producer connection has a consumer count of 0.  In this situation, the
> producer is still trying to send messages, but they never get to the
> consumer.
>
> Does anyone have advice on how to debug this?  We've turned on debug, but
> its really noisy and we are not sure if there are specific messages or
> loggers that we should be looking for.
>
> We have seen some errors on other brokers when a rebooted system comes
back
> and tries to connect again (this was when the broker wasn't shut down
> nicely).  They other brokers seem to think they still have a connection to
> that broker (no TCP SYN sent from rebooted broker I assume?).  I'm
> wondering, could it be that there is some sort of race condition when the
> original connection is cleaned up while a new one is created, and perhaps
> the new advertisement for consumer interest is cleaned up with it?
>
> A few things we have noticed that seem to contribute to the problem:
> * The AMQ service was not setup correctly to stop nicely on reboot, so it
> was likely just killed.  When this was corrected, we had difficulty
> reproducing the problem.
> * It seems limited to queues the consumer creates manually using an
> ActiveMQConnectionFactory.  The camel queues don't seem affected (sorry,
> not
> too familiar with camel yet).
>
>
> The ActiveMQConnectionFactory is created like this:
>
>
>
> Then a connection pool is created with it ("rawFactory" is the factory
> created above):
>
>
>
>
> This is a broker config from one of the servers.  They are all the same,
> except for the tweak to the networkConnectors list.
>
>
>
>
> Any hints or tips would be greatly appreciated!
>
> Thank you,
> Frank
>
>
>
> --
> View this message in context:
>
http://activemq.2283324.n4.nabble.com/Zero-consumers-on-some-brokers-after-reboot-tp4713319.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message