activemq-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dumitru HUSLEAG (JIRA)" <>
Subject [jira] [Issue Comment Deleted] (AMQ-6062) Broker goes 100% CPU on multi-queue command line browse
Date Wed, 02 Dec 2015 17:41:11 GMT


Dumitru HUSLEAG updated AMQ-6062:
    Comment: was deleted

(was: Thank you very much. I tested your fix with HotSpot and IBM and it's OK.
In case you want to add the test on Git repo you'll have to work on it because it is not deterministic
depending on timeout and browse task count parameters when the timeout arrives before all
browse finish. It needs to be fixed to be fast enough but nut so fast :) (to produce the bug),
and deterministic.)

> Broker goes 100% CPU on multi-queue command line browse
> -------------------------------------------------------
>                 Key: AMQ-6062
>                 URL:
>             Project: ActiveMQ
>          Issue Type: Bug
>          Components: Broker
>    Affects Versions: 5.9.1, 5.12.0
>         Environment: Linux Ubuntu, Aix, with both Java 6 and Java 7 (either IBM or Oracle)
>            Reporter: Dumitru HUSLEAG
>            Assignee: Christopher L. Shannon
>             Fix For: 5.12.2, 5.13.1, 5.14.0
>         Attachments:, ActiveMQ_BrowseInfiniteLoopUseCase.jmx, HotSpotThreadDump.txt,
IBMjava7AndAMQMaster.javacore.20151202.161643.18441.0001.txt, ThreadStatusAnalysis_2015-11-25_14h13m33s.png,
>   Original Estimate: 24h
>  Remaining Estimate: 24h
> Hello,
> The Broker may enter a state of high CPU consumption (100% or 200% if 2 browse enter
the infinite loop in the same time on a multi-core) when a command line browse on multiple
queues is issued. When that arrives the only way we found to restore the situation was to
kill the broker and restart because he was not responding anymore to clients. Killing the
clients and the browse process does not lower the CPU usage of the broker.
> I assume the infinite loop based on successive thread dumps that show the browse thread
remaines in the same code zone:
> Here is an example of the Browse thread call stack trace, when Broker is in high CPU
usage state:
> {quote}
> at java/util/HashMap.rehash( Code))
> at java/util/HashMap.rehash( Code))
> at java/util/HashMap.putImpl( Code))
> at java/util/HashMap.put( Code))
> at org/apache/activemq/broker/region/*QueueBrowserSubscription.isDuplicate(*(Compiled
> at org/apache/activemq/broker/region/*Queue.iterate(*(Compiled Code))
> at org/apache/activemq/thread/PooledTaskRunner.runTask(
> {quote}
> Tested ActiveMQ versions: 5.9.1 and the last, 5.12.0, but of course I suppose all in
beetwen are affected.
> The cause seems to be the non-threadsafe usage of the *audit HashMap* member of **
> Here are some facts on non thread-safe HashMap use problems:
> *High CPU / Hang on java.util.HashMap.findNonNullKeyEntry() due to non-thread-safe usage
of HashMap*
> I must underline that *this is not an IBM JDK specific problem as the IBM JDK is based
on Oracle JDK* and changing the JDK or JRE is not a solution.
> The scenario may be produced with a fresh non modified install (default config) of ActiveMQ
> 1. set ACTIVEMQ_HOME, JAVA_HOME and PATH to point accordingly to the right activemq,
and Java version.
> 2. cd $ACTIVEMQ_HOME/bin; ./activemq start
> 3. inject at least 2000 messages in each of two ques, say QA.ONE and QA.TWO
> 4. $ $ACTIVEMQ_HOME/bin/activemq browse --view JMSDestination,JMSMessageID --amqurl tcp://localhost:61616
> The simplest fix that I suggest is to change in the ** class
the type of *audit* member to *ConcurrentHashMap*.

This message was sent by Atlassian JIRA

View raw message