Return-Path: X-Original-To: apmail-activemq-dev-archive@www.apache.org Delivered-To: apmail-activemq-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id F3AC91765F for ; Thu, 9 Oct 2014 17:50:34 +0000 (UTC) Received: (qmail 46243 invoked by uid 500); 9 Oct 2014 17:50:34 -0000 Delivered-To: apmail-activemq-dev-archive@activemq.apache.org Received: (qmail 46134 invoked by uid 500); 9 Oct 2014 17:50:34 -0000 Mailing-List: contact dev-help@activemq.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@activemq.apache.org Delivered-To: mailing list dev@activemq.apache.org Received: (qmail 45866 invoked by uid 99); 9 Oct 2014 17:50:34 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Oct 2014 17:50:34 +0000 Date: Thu, 9 Oct 2014 17:50:34 +0000 (UTC) From: "Ilya (JIRA)" To: dev@activemq.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (AMQCPP-553) Deadlock under stress MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/AMQCPP-553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ilya updated AMQCPP-553: ------------------------ Description: Please have a look at the attached stress test. It consists of 12 reader threads, that create and destroy cms::MessageConsumers in a loop, and 4 writer threads, that send cms::TextMessages in a loop. Reader threads deadlock less than in 1 minute on my machine. To run it, simply issue {quote} make test {quote} The code depends on boost. I traced the issue down to inconsistent mutex acquisition order by the following two threads: {quote} thread 7 (Thread 0x7fa691fce700 (LWP 28088)) (gdb) bt #0 0x00007fa68eb405bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fa68fea1c06 in (anonymous namespace)::doMonitorEnter (monitor=0x7fa668007ff0, thread=0x7fa64c0071d0) at decaf/internal/util/concurrent/Threading.cpp:664 #2 0x00007fa68ff15dcb in decaf::util::concurrent::Lock::lock (this=0x7fa691fcd6b0) at decaf/util/concurrent/Lock.cpp:54 #3 0x00007fa68ff15ee5 in decaf::util::concurrent::Lock::Lock (this=, object=, intiallyLocked=) at decaf/util/concurrent/Lock.cpp:32 #4 0x00007fa68fc4da38 in activemq::core::kernels::ActiveMQConsumerKernel::dispatch (this=0x7fa680010a20, dispatch=...) at activemq/core/kernels/ActiveMQConsumerKernel.cpp:1527 #5 0x00007fa68fc06584 in activemq::core::ActiveMQSessionExecutor::dispatch (this=0x7fa65c005300, dispatch=...) at activemq/core/ActiveMQSessionExecutor.cpp:156 #6 0x00007fa68fc06f15 in activemq::core::ActiveMQSessionExecutor::iterate (this=0x7fa65c005300) at activemq/core/ActiveMQSessionExecutor.cpp:181 #7 0x00007fa68fd3fdf5 in activemq::threads::DedicatedTaskRunner::run (this=0x7fa64c004ab0) at activemq/threads/DedicatedTaskRunner.cpp:141 Waiting for internal->listenerMutex, that is held by thread 6 Acquisition order: ActiveMQConsumerKernel::internal->unconsumedMessages ActiveMQConsumerKernel::internal->listenerMutex (gdb) frame 1 (gdb) p/x monitor->owner->handle $9 = 0x7fa691fbd700 thread 6 (Thread 0x7fa691fbd700 (LWP 28091)) (gdb) bt #0 0x00007fa68eb405bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fa68fea1c06 in (anonymous namespace)::doMonitorEnter (monitor=0x7fa6680071e0, thread=0x7fa6246a13d0) at decaf/internal/util/concurrent/Threading.cpp:664 #2 0x00007fa68ff15dcb in decaf::util::concurrent::Lock::lock (this=0x7fa691fbc6f0) at decaf/util/concurrent/Lock.cpp:54 #3 0x00007fa68ff15ee5 in decaf::util::concurrent::Lock::Lock (this=, object=, intiallyLocked=) at decaf/util/concurrent/Lock.cpp:32 #4 0x00007fa68fc3c21a in activemq::core::SimplePriorityMessageDispatchChannel::dequeueNoWait (this=0x7fa68000eee0) at activemq/core/SimplePriorityMessageDispatchChannel.cpp:95 #5 0x00007fa68fc40f3c in activemq::core::kernels::ActiveMQConsumerKernel::iterate (this=0x7fa680010a20) at activemq/core/kernels/ActiveMQConsumerKernel.cpp:1701 #6 0x00007fa68fc8ab23 in activemq::core::kernels::ActiveMQSessionKernel::iterateConsumers (this=0x7fa65c003960) at activemq/core/kernels/ActiveMQSessionKernel.cpp:1370 #7 0x00007fa68fc06eb9 in activemq::core::ActiveMQSessionExecutor::iterate (this=0x7fa65c005300) at activemq/core/ActiveMQSessionExecutor.cpp:173 #8 0x00007fa68fd3fdf5 in activemq::threads::DedicatedTaskRunner::run (this=0x7fa624603e00) at activemq/threads/DedicatedTaskRunner.cpp:141 Waiting for mutex, that is held by thread 7 Acquisition order: ActiveMQSessionKernel::config->consumerLock ActiveMQConsumerKernel::internal->listenerMutex ActiveMQConsumerKernel::internal->unconsumedMessages (gdb) frame 1 (gdb) p/x monitor->owner->handle $10 = 0x7fa691fce700 {quote} was: Please have a look at the attached stress test. It consists of 12 reader threads, that create and destroy cms::MessageConsumers in a loop, and 4 writer threads, that send cms::TextMessages in a loop. Reader threads deadlock less than in 1 minute on my machine. To run it, simply issue bq. make test The code depends on boost. I traced the issue down to inconsistent mutex acquisition order by the following two threads: bq. thread 7 (Thread 0x7fa691fce700 (LWP 28088)) (gdb) bt #0 0x00007fa68eb405bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fa68fea1c06 in (anonymous namespace)::doMonitorEnter (monitor=0x7fa668007ff0, thread=0x7fa64c0071d0) at decaf/internal/util/concurrent/Threading.cpp:664 #2 0x00007fa68ff15dcb in decaf::util::concurrent::Lock::lock (this=0x7fa691fcd6b0) at decaf/util/concurrent/Lock.cpp:54 #3 0x00007fa68ff15ee5 in decaf::util::concurrent::Lock::Lock (this=, object=, intiallyLocked=) at decaf/util/concurrent/Lock.cpp:32 #4 0x00007fa68fc4da38 in activemq::core::kernels::ActiveMQConsumerKernel::dispatch (this=0x7fa680010a20, dispatch=...) at activemq/core/kernels/ActiveMQConsumerKernel.cpp:1527 #5 0x00007fa68fc06584 in activemq::core::ActiveMQSessionExecutor::dispatch (this=0x7fa65c005300, dispatch=...) at activemq/core/ActiveMQSessionExecutor.cpp:156 #6 0x00007fa68fc06f15 in activemq::core::ActiveMQSessionExecutor::iterate (this=0x7fa65c005300) at activemq/core/ActiveMQSessionExecutor.cpp:181 #7 0x00007fa68fd3fdf5 in activemq::threads::DedicatedTaskRunner::run (this=0x7fa64c004ab0) at activemq/threads/DedicatedTaskRunner.cpp:141 Waiting for internal->listenerMutex, that is held by thread 6 Acquisition order: ActiveMQConsumerKernel::internal->unconsumedMessages ActiveMQConsumerKernel::internal->listenerMutex (gdb) frame 1 (gdb) p/x monitor->owner->handle $9 = 0x7fa691fbd700 thread 6 (Thread 0x7fa691fbd700 (LWP 28091)) (gdb) bt #0 0x00007fa68eb405bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fa68fea1c06 in (anonymous namespace)::doMonitorEnter (monitor=0x7fa6680071e0, thread=0x7fa6246a13d0) at decaf/internal/util/concurrent/Threading.cpp:664 #2 0x00007fa68ff15dcb in decaf::util::concurrent::Lock::lock (this=0x7fa691fbc6f0) at decaf/util/concurrent/Lock.cpp:54 #3 0x00007fa68ff15ee5 in decaf::util::concurrent::Lock::Lock (this=, object=, intiallyLocked=) at decaf/util/concurrent/Lock.cpp:32 #4 0x00007fa68fc3c21a in activemq::core::SimplePriorityMessageDispatchChannel::dequeueNoWait (this=0x7fa68000eee0) at activemq/core/SimplePriorityMessageDispatchChannel.cpp:95 #5 0x00007fa68fc40f3c in activemq::core::kernels::ActiveMQConsumerKernel::iterate (this=0x7fa680010a20) at activemq/core/kernels/ActiveMQConsumerKernel.cpp:1701 #6 0x00007fa68fc8ab23 in activemq::core::kernels::ActiveMQSessionKernel::iterateConsumers (this=0x7fa65c003960) at activemq/core/kernels/ActiveMQSessionKernel.cpp:1370 #7 0x00007fa68fc06eb9 in activemq::core::ActiveMQSessionExecutor::iterate (this=0x7fa65c005300) at activemq/core/ActiveMQSessionExecutor.cpp:173 #8 0x00007fa68fd3fdf5 in activemq::threads::DedicatedTaskRunner::run (this=0x7fa624603e00) at activemq/threads/DedicatedTaskRunner.cpp:141 Waiting for mutex, that is held by thread 7 Acquisition order: ActiveMQSessionKernel::config->consumerLock ActiveMQConsumerKernel::internal->listenerMutex ActiveMQConsumerKernel::internal->unconsumedMessages (gdb) frame 1 (gdb) p/x monitor->owner->handle $10 = 0x7fa691fce700 > Deadlock under stress > --------------------- > > Key: AMQCPP-553 > URL: https://issues.apache.org/jira/browse/AMQCPP-553 > Project: ActiveMQ C++ Client > Issue Type: Bug > Components: CMS Impl > Affects Versions: 3.8.3 > Environment: x86_64 > Linux 3.13.0-36 > gcc 4.4.7 > Reporter: Ilya > Assignee: Timothy Bish > Attachments: amqcpp-deadlock.tar.xz > > > Please have a look at the attached stress test. It consists of 12 reader threads, that create and destroy cms::MessageConsumers in a loop, and 4 writer threads, that send cms::TextMessages in a loop. Reader threads deadlock less than in 1 minute on my machine. To run it, simply issue > {quote} > make test > {quote} > The code depends on boost. > I traced the issue down to inconsistent mutex acquisition order by the following two threads: > {quote} > thread 7 (Thread 0x7fa691fce700 (LWP 28088)) > (gdb) bt > #0 0x00007fa68eb405bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 > #1 0x00007fa68fea1c06 in (anonymous namespace)::doMonitorEnter (monitor=0x7fa668007ff0, thread=0x7fa64c0071d0) at decaf/internal/util/concurrent/Threading.cpp:664 > #2 0x00007fa68ff15dcb in decaf::util::concurrent::Lock::lock (this=0x7fa691fcd6b0) at decaf/util/concurrent/Lock.cpp:54 > #3 0x00007fa68ff15ee5 in decaf::util::concurrent::Lock::Lock (this=, object=, intiallyLocked=) > at decaf/util/concurrent/Lock.cpp:32 > #4 0x00007fa68fc4da38 in activemq::core::kernels::ActiveMQConsumerKernel::dispatch (this=0x7fa680010a20, dispatch=...) at activemq/core/kernels/ActiveMQConsumerKernel.cpp:1527 > #5 0x00007fa68fc06584 in activemq::core::ActiveMQSessionExecutor::dispatch (this=0x7fa65c005300, dispatch=...) at activemq/core/ActiveMQSessionExecutor.cpp:156 > #6 0x00007fa68fc06f15 in activemq::core::ActiveMQSessionExecutor::iterate (this=0x7fa65c005300) at activemq/core/ActiveMQSessionExecutor.cpp:181 > #7 0x00007fa68fd3fdf5 in activemq::threads::DedicatedTaskRunner::run (this=0x7fa64c004ab0) at activemq/threads/DedicatedTaskRunner.cpp:141 > Waiting for internal->listenerMutex, that is held by thread 6 > Acquisition order: > ActiveMQConsumerKernel::internal->unconsumedMessages > ActiveMQConsumerKernel::internal->listenerMutex > (gdb) frame 1 > (gdb) p/x monitor->owner->handle > $9 = 0x7fa691fbd700 > thread 6 (Thread 0x7fa691fbd700 (LWP 28091)) > (gdb) bt > #0 0x00007fa68eb405bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 > #1 0x00007fa68fea1c06 in (anonymous namespace)::doMonitorEnter (monitor=0x7fa6680071e0, thread=0x7fa6246a13d0) at decaf/internal/util/concurrent/Threading.cpp:664 > #2 0x00007fa68ff15dcb in decaf::util::concurrent::Lock::lock (this=0x7fa691fbc6f0) at decaf/util/concurrent/Lock.cpp:54 > #3 0x00007fa68ff15ee5 in decaf::util::concurrent::Lock::Lock (this=, object=, intiallyLocked=) > at decaf/util/concurrent/Lock.cpp:32 > #4 0x00007fa68fc3c21a in activemq::core::SimplePriorityMessageDispatchChannel::dequeueNoWait (this=0x7fa68000eee0) at activemq/core/SimplePriorityMessageDispatchChannel.cpp:95 > #5 0x00007fa68fc40f3c in activemq::core::kernels::ActiveMQConsumerKernel::iterate (this=0x7fa680010a20) at activemq/core/kernels/ActiveMQConsumerKernel.cpp:1701 > #6 0x00007fa68fc8ab23 in activemq::core::kernels::ActiveMQSessionKernel::iterateConsumers (this=0x7fa65c003960) at activemq/core/kernels/ActiveMQSessionKernel.cpp:1370 > #7 0x00007fa68fc06eb9 in activemq::core::ActiveMQSessionExecutor::iterate (this=0x7fa65c005300) at activemq/core/ActiveMQSessionExecutor.cpp:173 > #8 0x00007fa68fd3fdf5 in activemq::threads::DedicatedTaskRunner::run (this=0x7fa624603e00) at activemq/threads/DedicatedTaskRunner.cpp:141 > Waiting for mutex, that is held by thread 7 > Acquisition order: > ActiveMQSessionKernel::config->consumerLock > ActiveMQConsumerKernel::internal->listenerMutex > ActiveMQConsumerKernel::internal->unconsumedMessages > (gdb) frame 1 > (gdb) p/x monitor->owner->handle > $10 = 0x7fa691fce700 > {quote} -- This message was sent by Atlassian JIRA (v6.3.4#6332)