Return-Path: X-Original-To: apmail-activemq-issues-archive@minotaur.apache.org Delivered-To: apmail-activemq-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 8E26A1810A for ; Wed, 25 Nov 2015 13:32:16 +0000 (UTC) Received: (qmail 96676 invoked by uid 500); 25 Nov 2015 13:32:16 -0000 Delivered-To: apmail-activemq-issues-archive@activemq.apache.org Received: (qmail 96488 invoked by uid 500); 25 Nov 2015 13:32:16 -0000 Mailing-List: contact issues-help@activemq.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@activemq.apache.org Delivered-To: mailing list issues@activemq.apache.org Received: (qmail 95942 invoked by uid 99); 25 Nov 2015 13:32:12 -0000 Received: from arcas.apache.org (HELO arcas) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 Nov 2015 13:32:12 +0000 Received: from arcas.apache.org (localhost [127.0.0.1]) by arcas (Postfix) with ESMTP id 2D7DF2C1F89 for ; Wed, 25 Nov 2015 13:32:11 +0000 (UTC) Date: Wed, 25 Nov 2015 13:32:11 +0000 (UTC) From: "Dumitru HUSLEAG (JIRA)" To: issues@activemq.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (AMQ-6062) Broker goes 100% CPU on multi-queue command line browse MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/AMQ-6062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dumitru HUSLEAG updated AMQ-6062: --------------------------------- Description: Hello, The Broker may enter a state of high CPU consumption (100% or 200% if 2 browse enter the infinite loop in the same time on a multi-core) when a command line brose on multiple ques is issued. When that arives the only way we found to restore the situation was to kill the broker and restart because he was not responding anymore to clients. Killing the clients and the browse process does not lower the CPU usage of the broker. I assume the infinite loop based on successive thread dumps that show the browse thread remaines in the same code zone: Another IBM Link: http://www-01.ibm.com/support/docview.wss?uid=swg21597581 {quote} at java/util/HashMap.rehash(HashMap.java:782(Compiled Code)) at java/util/HashMap.rehash(HashMap.java:819(Compiled Code)) at java/util/HashMap.putImpl(HashMap.java:702(Compiled Code)) at java/util/HashMap.put(HashMap.java:680(Compiled Code)) at org/apache/activemq/broker/region/*QueueBrowserSubscription.isDuplicate(QueueBrowserSubscription.java:72*(Compiled Code)) at org/apache/activemq/broker/region/*Queue.iterate(Queue.java:1688*(Compiled Code)) at org/apache/activemq/thread/PooledTaskRunner.runTask(PooledTaskRunner.java:133(Compiled Code)). {quote} Tested versions: 5.9.1 and the last, 5.12.0, but of course I suppose all in beetwen are affected. The cause seems to be the non-threadsafe usage of the audit HashMap member of QueueBrowserSubscription.java class. The scenario may be produced with a fresh non modified install (default config) of ActiveMQ broker: 1. set ACTIVEMQ_HOME, JAVA_HOME and PATH to point accordingly to the right activemq, and Java version. 2. cd $ACTIVEMQ_HOME/bin; ./activemq start 3. inject at least 2000 messages in each of two ques, say QA.ONE and QA.TWO 4. $ $ACTIVEMQ_HOME/bin/activemq browse --view JMSDestination,JMSMessageID --amqurl tcp://localhost:61616 QA.* was: Hello, The Broker may enter a state of high CPU consumption (100% or 200% if 2 browse enter the infinite loop in the same time on a multi-core) when a command line brose on multiple ques is issued. When that arives the only way we found to restore the situation was to kill the broker and restart because he was not responding anymore to clients. Killing the clients and the browse process does not lower the CPU usage of the broker. I assume the infinite loop based on successive thread dumps that show the browse thread remaines in the same code zone: Another IBM Link: http://www-01.ibm.com/support/docview.wss?uid=swg21597581 {quote} at java/util/HashMap.rehash(HashMap.java:782(Compiled Code)) at java/util/HashMap.rehash(HashMap.java:819(Compiled Code)) at java/util/HashMap.putImpl(HashMap.java:702(Compiled Code)) at java/util/HashMap.put(HashMap.java:680(Compiled Code)) *at org/apache/activemq/broker/region/QueueBrowserSubscription.isDuplicate(QueueBrowserSubscription.java:72(Compiled Code)) at org/apache/activemq/broker/region/Queue.iterate(Queue.java:1688(Compiled Code))* at org/apache/activemq/thread/PooledTaskRunner.runTask(PooledTaskRunner.java:133(Compiled Code)). {quote} Tested versions: 5.9.1 and the last, 5.12.0, but of course I suppose all in beetwen are affected. The cause seems to be the non-threadsafe usage of the audit HashMap member of QueueBrowserSubscription.java class. The scenario may be produced with a fresh non modified install (default config) of ActiveMQ broker: 1. set ACTIVEMQ_HOME, JAVA_HOME and PATH to point accordingly to the right activemq, and Java version. 2. cd $ACTIVEMQ_HOME/bin; ./activemq start 3. inject at least 2000 messages in each of two ques, say QA.ONE and QA.TWO 4. $ $ACTIVEMQ_HOME/bin/activemq browse --view JMSDestination,JMSMessageID --amqurl tcp://localhost:61616 QA.* > Broker goes 100% CPU on multi-queue command line browse > ------------------------------------------------------- > > Key: AMQ-6062 > URL: https://issues.apache.org/jira/browse/AMQ-6062 > Project: ActiveMQ > Issue Type: Bug > Components: Broker > Affects Versions: 5.9.1, 5.12.0 > Environment: Linux Ubuntu, Aix, with both Java 6 and Java 7 (either IBM or Oracle) > Reporter: Dumitru HUSLEAG > Priority: Critical > Attachments: ActiveMQ_BrowseInfiniteLoopUseCase.jmx, ThreadStatusAnalysis_2015-11-25_14h13m33s.png > > Original Estimate: 24h > Remaining Estimate: 24h > > Hello, > The Broker may enter a state of high CPU consumption (100% or 200% if 2 browse enter the infinite loop in the same time on a multi-core) when a command line brose on multiple ques is issued. When that arives the only way we found to restore the situation was to kill the broker and restart because he was not responding anymore to clients. Killing the clients and the browse process does not lower the CPU usage of the broker. > I assume the infinite loop based on successive thread dumps that show the browse thread remaines in the same code zone: > Another IBM Link: > http://www-01.ibm.com/support/docview.wss?uid=swg21597581 > {quote} > at java/util/HashMap.rehash(HashMap.java:782(Compiled Code)) > at java/util/HashMap.rehash(HashMap.java:819(Compiled Code)) > at java/util/HashMap.putImpl(HashMap.java:702(Compiled Code)) > at java/util/HashMap.put(HashMap.java:680(Compiled Code)) > at org/apache/activemq/broker/region/*QueueBrowserSubscription.isDuplicate(QueueBrowserSubscription.java:72*(Compiled Code)) > at org/apache/activemq/broker/region/*Queue.iterate(Queue.java:1688*(Compiled Code)) > at org/apache/activemq/thread/PooledTaskRunner.runTask(PooledTaskRunner.java:133(Compiled Code)). > {quote} > Tested versions: 5.9.1 and the last, 5.12.0, but of course I suppose all in beetwen are affected. > The cause seems to be the non-threadsafe usage of the audit HashMap member of QueueBrowserSubscription.java class. > The scenario may be produced with a fresh non modified install (default config) of ActiveMQ broker: > 1. set ACTIVEMQ_HOME, JAVA_HOME and PATH to point accordingly to the right activemq, and Java version. > 2. cd $ACTIVEMQ_HOME/bin; ./activemq start > 3. inject at least 2000 messages in each of two ques, say QA.ONE and QA.TWO > 4. $ $ACTIVEMQ_HOME/bin/activemq browse --view JMSDestination,JMSMessageID --amqurl tcp://localhost:61616 QA.* -- This message was sent by Atlassian JIRA (v6.3.4#6332)