activemq-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dumitru HUSLEAG (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (AMQ-6062) Broker goes 100% CPU on multi-queue command line browse
Date Wed, 25 Nov 2015 13:32:11 GMT

     [ https://issues.apache.org/jira/browse/AMQ-6062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Dumitru HUSLEAG updated AMQ-6062:
---------------------------------
    Description: 
Hello,
The Broker may enter a state of high CPU consumption (100% or 200% if 2 browse enter the infinite
loop in the same time on a multi-core) when a command line brose on multiple ques is issued.
When that arives the only way we found to restore the situation was to kill the broker and
restart because he was not responding anymore to clients. Killing the clients and the browse
process does not lower the CPU usage of the broker.
I assume the infinite loop based on successive thread dumps that show the browse thread remaines
in the same code zone:

Another IBM Link:
http://www-01.ibm.com/support/docview.wss?uid=swg21597581

{quote}
at java/util/HashMap.rehash(HashMap.java:782(Compiled Code))
at java/util/HashMap.rehash(HashMap.java:819(Compiled Code))
at java/util/HashMap.putImpl(HashMap.java:702(Compiled Code))
at java/util/HashMap.put(HashMap.java:680(Compiled Code))
at org/apache/activemq/broker/region/*QueueBrowserSubscription.isDuplicate(QueueBrowserSubscription.java:72*(Compiled
Code))
at org/apache/activemq/broker/region/*Queue.iterate(Queue.java:1688*(Compiled Code))
at org/apache/activemq/thread/PooledTaskRunner.runTask(PooledTaskRunner.java:133(Compiled
Code)).
{quote}

Tested versions: 5.9.1 and the last, 5.12.0, but of course I suppose all in beetwen are affected.

The cause seems to be the non-threadsafe usage of the audit HashMap member of QueueBrowserSubscription.java
class.

The scenario may be produced with a fresh non modified install (default config) of ActiveMQ
broker:
1. set ACTIVEMQ_HOME, JAVA_HOME and PATH to point accordingly to the right activemq, and Java
version.
2. cd $ACTIVEMQ_HOME/bin; ./activemq start
3. inject at least 2000 messages in each of two ques, say QA.ONE and QA.TWO
4. $ $ACTIVEMQ_HOME/bin/activemq browse --view JMSDestination,JMSMessageID --amqurl tcp://localhost:61616
QA.*

  was:
Hello,
The Broker may enter a state of high CPU consumption (100% or 200% if 2 browse enter the infinite
loop in the same time on a multi-core) when a command line brose on multiple ques is issued.
When that arives the only way we found to restore the situation was to kill the broker and
restart because he was not responding anymore to clients. Killing the clients and the browse
process does not lower the CPU usage of the broker.
I assume the infinite loop based on successive thread dumps that show the browse thread remaines
in the same code zone:

Another IBM Link:
http://www-01.ibm.com/support/docview.wss?uid=swg21597581

{quote}
at java/util/HashMap.rehash(HashMap.java:782(Compiled Code))
at java/util/HashMap.rehash(HashMap.java:819(Compiled Code))
at java/util/HashMap.putImpl(HashMap.java:702(Compiled Code))
at java/util/HashMap.put(HashMap.java:680(Compiled Code))
*at org/apache/activemq/broker/region/QueueBrowserSubscription.isDuplicate(QueueBrowserSubscription.java:72(Compiled
Code))
at org/apache/activemq/broker/region/Queue.iterate(Queue.java:1688(Compiled Code))*
at org/apache/activemq/thread/PooledTaskRunner.runTask(PooledTaskRunner.java:133(Compiled
Code)).
{quote}

Tested versions: 5.9.1 and the last, 5.12.0, but of course I suppose all in beetwen are affected.

The cause seems to be the non-threadsafe usage of the audit HashMap member of QueueBrowserSubscription.java
class.

The scenario may be produced with a fresh non modified install (default config) of ActiveMQ
broker:
1. set ACTIVEMQ_HOME, JAVA_HOME and PATH to point accordingly to the right activemq, and Java
version.
2. cd $ACTIVEMQ_HOME/bin; ./activemq start
3. inject at least 2000 messages in each of two ques, say QA.ONE and QA.TWO
4. $ $ACTIVEMQ_HOME/bin/activemq browse --view JMSDestination,JMSMessageID --amqurl tcp://localhost:61616
QA.*


> Broker goes 100% CPU on multi-queue command line browse
> -------------------------------------------------------
>
>                 Key: AMQ-6062
>                 URL: https://issues.apache.org/jira/browse/AMQ-6062
>             Project: ActiveMQ
>          Issue Type: Bug
>          Components: Broker
>    Affects Versions: 5.9.1, 5.12.0
>         Environment: Linux Ubuntu, Aix, with both Java 6 and Java 7 (either IBM or Oracle)
>            Reporter: Dumitru HUSLEAG
>            Priority: Critical
>         Attachments: ActiveMQ_BrowseInfiniteLoopUseCase.jmx, ThreadStatusAnalysis_2015-11-25_14h13m33s.png
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> Hello,
> The Broker may enter a state of high CPU consumption (100% or 200% if 2 browse enter
the infinite loop in the same time on a multi-core) when a command line brose on multiple
ques is issued. When that arives the only way we found to restore the situation was to kill
the broker and restart because he was not responding anymore to clients. Killing the clients
and the browse process does not lower the CPU usage of the broker.
> I assume the infinite loop based on successive thread dumps that show the browse thread
remaines in the same code zone:
> Another IBM Link:
> http://www-01.ibm.com/support/docview.wss?uid=swg21597581
> {quote}
> at java/util/HashMap.rehash(HashMap.java:782(Compiled Code))
> at java/util/HashMap.rehash(HashMap.java:819(Compiled Code))
> at java/util/HashMap.putImpl(HashMap.java:702(Compiled Code))
> at java/util/HashMap.put(HashMap.java:680(Compiled Code))
> at org/apache/activemq/broker/region/*QueueBrowserSubscription.isDuplicate(QueueBrowserSubscription.java:72*(Compiled
Code))
> at org/apache/activemq/broker/region/*Queue.iterate(Queue.java:1688*(Compiled Code))
> at org/apache/activemq/thread/PooledTaskRunner.runTask(PooledTaskRunner.java:133(Compiled
Code)).
> {quote}
> Tested versions: 5.9.1 and the last, 5.12.0, but of course I suppose all in beetwen are
affected.
> The cause seems to be the non-threadsafe usage of the audit HashMap member of QueueBrowserSubscription.java
class.
> The scenario may be produced with a fresh non modified install (default config) of ActiveMQ
broker:
> 1. set ACTIVEMQ_HOME, JAVA_HOME and PATH to point accordingly to the right activemq,
and Java version.
> 2. cd $ACTIVEMQ_HOME/bin; ./activemq start
> 3. inject at least 2000 messages in each of two ques, say QA.ONE and QA.TWO
> 4. $ $ACTIVEMQ_HOME/bin/activemq browse --view JMSDestination,JMSMessageID --amqurl tcp://localhost:61616
QA.*



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message