activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pete Bertrand (JIRA)" <>
Subject [jira] [Created] (AMQ-5424) Broker at 100% CPU when idle after Network Connection reconnect with duplicates sent
Date Thu, 06 Nov 2014 01:10:34 GMT
Pete Bertrand created AMQ-5424:

             Summary: Broker at 100% CPU when idle after Network Connection reconnect with
duplicates sent
                 Key: AMQ-5424
             Project: ActiveMQ
          Issue Type: Bug
    Affects Versions: 5.10.0
            Reporter: Pete Bertrand

In a network of 2 brokers (A and B) with durable queued messages 
going from A to B over a duplex NetworkConnector,
if A is stopped and restarted while messages are in-flight, 
and if replayed messages from A are recognized as duplicates on B,
then 30 seconds after B goes idle, B's CPU goes to 100%.

I have attached the thread dump to the ticket.

>From what I have been able to figure out, the dequeue counter does not count
moving the duplicate into the DLQ. The counters show a pending message when
there is none in the persisted queue. So when the scheduler kicks in 30 seconds
after the broker goes idle, it says "I have a pending message, fetch it from the DB"
but the fetch returns 0 messages. Immediately the scheduler still sees pending
messages and does a DB fetch, with no results. This is where the CPU is spinning.

See the attached thread dump.

So, in detail:
It appears that after A is restarted and it replays messages that have not been ACKed,
B receives duplicate messages and sends them to the DLQ. Here is the warning from the log:
  WARN | duplicate message from store ID:host-lnx-59946-1415221396197-1:1:1:1:468, redirecting
for dlq processing | | ActiveMQ VMTransport: vm://broker1#11-1
After all messages are delivered and the brokers are idle for 30 seconds and the CPU on B
is now 100%, if you use the WebConsole and look at the queues on B you see the following:
              Number Of                	
   Queue      Pending     Number Of  Messages  Messages
   Name       Messages    Consumers  Enqueued  Dequeued
ActiveMQ.DLQ     1            0         1         0
TEST.FOO         1            1        469       468

On this test run, only one message was a duplicate. It was moved to the DLQ, but the TEST.FOO
counters show it as pending. The counters are out of sync with actual messages in the persisted
queue, because the duplicate message is now in the DLQ and not in the TEST.FOO queue.

At this point if you purge TEST.FOO, CPU on B goes back to normal because this clears the
pending message counter.

+*Steps to reproduce*+

Set up 2 brokers as follows:

  *producer* ==> *broker-A*  <==  duplex network connection  ==>  *broker-B* ==>

1) Download the binary distribution of AMQ 5.10.0 and extract apache-activemq-5.10.0-bin.tar.gz

2) Create two brokers
 $ ACTIVEMQ_HOME/bin/activemq create /path/to/brokers/broker-a
 $ ACTIVEMQ_HOME/bin/activemq create /path/to/brokers/broker-b

3) Update broker-a to connect to broker-b with a duplex connection.
   _You can use the attached *activemq.xml*_. It does the following:
- Sets transport for broker-a to port 61610
- Sets up networkConnector to connect to broker-b on 61616
- Does not start jetty web console on broker-a to avoid port conflict

broker-b is un-modified and defaults to port 61616

4) Start the brokers
 $ broker-a/bin/broker-a start
 $ broker-b/bin/broker-b start

5) Start consumer connected to broker-b and producer connected to broker-a
 $ ant consumer -Durl=tcp://localhost:61616 -Ddurable=true
 $ ant producer -Durl=tcp://localhost:61610 -Ddurable=true

6) Stop broker-a before producer is finished sending messages, then restart
 $ broker-a/bin/broker-a stop
 $ broker-a/bin/broker-a start

7) Look at broker-b logs for duplicates, look at broker-b web console for pending messages

8) 30 seconds after going idle, broker-b CPU will goto 100%

9) Purge TEST.FOO on broker-b, pending messages will reset and CPU will go back to normal.

This message was sent by Atlassian JIRA

View raw message