Return-Path: Delivered-To: apmail-activemq-users-archive@www.apache.org Received: (qmail 15524 invoked from network); 14 Mar 2011 16:53:28 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 14 Mar 2011 16:53:28 -0000 Received: (qmail 80748 invoked by uid 500); 14 Mar 2011 16:53:27 -0000 Delivered-To: apmail-activemq-users-archive@activemq.apache.org Received: (qmail 80708 invoked by uid 500); 14 Mar 2011 16:53:27 -0000 Mailing-List: contact users-help@activemq.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: users@activemq.apache.org Delivered-To: mailing list users@activemq.apache.org Received: (qmail 80700 invoked by uid 99); 14 Mar 2011 16:53:27 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Mar 2011 16:53:27 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: local policy) Received: from [209.85.161.43] (HELO mail-fx0-f43.google.com) (209.85.161.43) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Mar 2011 16:53:20 +0000 Received: by fxm3 with SMTP id 3so3856391fxm.2 for ; Mon, 14 Mar 2011 09:52:59 -0700 (PDT) Received: by 10.223.76.147 with SMTP id c19mr1708483fak.55.1300121552829; Mon, 14 Mar 2011 09:52:32 -0700 (PDT) MIME-Version: 1.0 Received: by 10.223.96.68 with HTTP; Mon, 14 Mar 2011 09:51:58 -0700 (PDT) In-Reply-To: References: From: Oscar Pernas Date: Mon, 14 Mar 2011 17:51:58 +0100 Message-ID: Subject: Re: ActiveMQ-CPP: deadlock when broker restarts and failover is used To: users@activemq.apache.org Cc: chris Content-Type: multipart/alternative; boundary=0015174c3fb0523d5b049e7423d5 X-Virus-Checked: Checked by ClamAV on apache.org --0015174c3fb0523d5b049e7423d5 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable This bug was reported some time ago. If you want to make a workaround, if you are consuming messages in asynchronous (by callback), change to synchronous consuming mode. In my case its resolved it. 2011/3/14 chris > Hi all, > > we seem to have a deadlock situation when a broker shutdown happens. > We're in the middle of onMessage callback and trying to call commit. > The ActiveMQConsumer::dispatch method holds the "unconsumedMessages" > lock and waits for the "reconnectMutex" when committing. The failover > transport calls clearMessagesInProgress which tries to get hold of the > "unconsumedMessages" lock but holds the "reconnectMutex" in > FailoverTransport::handleTransportFailure. The two stack traces below > should show what happens. > > We're using ActiveMQ-CPP 3.2.5 with a 5.3.2 broker. > > Kind regards, > Chris > > One thread here: > ntdll.dll!77dbf8c1() > [Frames below may be incorrect and/or missing, no symbols loaded > for > ntdll.dll] > ntdll.dll!77dbf8c1() > ntdll.dll!77dd8dd4() > kernel32.dll!775c14dd() > > > > activemq-cppd.dll!decaf::internal::util::concurrent::MutexImpl::lock(deca= f::util::concurrent::MutexHandle > * handle=3D0x039e0ed0) Line 68 + 0xa bytes C++ > activemq-cppd.dll!decaf::util::concurrent::Mutex::lock() Line 75 + > 0xe bytes C++ > > > activemq-cppd.dll!decaf::util::StlQueue > >::lock() Line 254 C++ > activemq-cppd.dll!activemq::core::MessageDispatchChannel::lock() > Line 154 C++ > activemq-cppd.dll!decaf::util::concurrent::Lock::lock() Line 55 > C++ > > > activemq-cppd.dll!decaf::util::concurrent::Lock::Lock(decaf::util::concur= rent::Synchronizable > * object=3D0x039d4c48, const bool intiallyLocked=3Dtrue) Line 34 C++ > > > activemq-cppd.dll!activemq::core::ActiveMQConsumer::clearMessagesInProgre= ss() > Line 1112 + 0x17 bytes C++ > > > activemq-cppd.dll!activemq::core::ActiveMQSession::clearMessagesInProgres= s() > Line 240 C++ > > > activemq-cppd.dll!activemq::core::ActiveMQConnection::transportInterrupte= d() > Line 705 C++ > > > activemq-cppd.dll!activemq::transport::TransportFilter::transportInterrup= ted() > Line 67 + 0x13 bytes C++ > > > activemq-cppd.dll!activemq::transport::failover::FailoverTransport::handl= eTransportFailure(const > decaf::lang::Exception & error=3D{...}) Line 479 C++ > > > activemq-cppd.dll!activemq::transport::failover::FailoverTransportListene= r::onException(const > decaf::lang::Exception & ex=3D{...}) Line 97 + 0xf bytes C++ > activemq-cppd.dll!activemq::transport::TransportFilter::fire(const > decaf::lang::Exception & ex=3D{...}) Line 49 + 0x17 bytes C++ > > activemq-cppd.dll!activemq::transport::TransportFilter::onException(const > decaf::lang::Exception & ex=3D{...}) Line 42 C++ > activemq-cppd.dll!activemq::transport::TransportFilter::fire(const > decaf::lang::Exception & ex=3D{...}) Line 49 + 0x17 bytes C++ > > activemq-cppd.dll!activemq::transport::TransportFilter::onException(const > decaf::lang::Exception & ex=3D{...}) Line 42 C++ > > > activemq-cppd.dll!activemq::transport::inactivity::InactivityMonitor::onE= xception(const > decaf::lang::Exception & ex=3D{...}) Line 276 C++ > activemq-cppd.dll!activemq::transport::TransportFilter::fire(const > decaf::lang::Exception & ex=3D{...}) Line 49 + 0x17 bytes C++ > > activemq-cppd.dll!activemq::transport::TransportFilter::onException(const > decaf::lang::Exception & ex=3D{...}) Line 42 C++ > > > activemq-cppd.dll!activemq::transport::IOTransport::fire(decaf::lang::Exc= eption > & ex=3D{...}) Line 73 + 0x17 bytes C++ > activemq-cppd.dll!activemq::transport::IOTransport::run() Line 247 > C++ > > > activemq-cppd.dll!decaf::lang::ThreadProperties::runCallback(decaf::lang:= :ThreadProperties > * properties=3D0x039cb968) Line 135 + 0x11 bytes C++ > activemq-cppd.dll!`anonymous namespace'::threadWorker(void * > arg=3D0x039cb968) Line 208 + 0x9 bytes C++ > msvcr100d.dll!_callthreadstartex() Line 314 + 0xf bytes C > msvcr100d.dll!_threadstartex(void * ptd=3D0x0396e030) Line 297 C > kernel32.dll!775c33ca() > ntdll.dll!77dd9ed2() > ntdll.dll!77dd9ea5() > > Dispatch thread here: > > ntdll.dll!77dbf8c1() > [Frames below may be incorrect and/or missing, no symbols loaded > for > ntdll.dll] > ntdll.dll!77dbf8c1() > ntdll.dll!77dd8dd4() > ntdll.dll!77dd8cb8() > > > > activemq-cppd.dll!decaf::internal::util::concurrent::MutexImpl::lock(deca= f::util::concurrent::MutexHandle > * handle=3D0x039b6988) Line 68 + 0xa bytes C++ > activemq-cppd.dll!decaf::util::concurrent::Mutex::lock() Line 75 + > 0xe bytes C++ > activemq-cppd.dll!decaf::util::concurrent::Lock::lock() Line 55 > C++ > > > activemq-cppd.dll!decaf::util::concurrent::Lock::Lock(decaf::util::concur= rent::Synchronizable > * object=3D0x039d4974, const bool intiallyLocked=3Dtrue) Line 34 C++ > > > activemq-cppd.dll!activemq::transport::failover::FailoverTransport::onewa= y(const > > decaf::lang::Pointer > & command=3D{...}) Line 186 + 0x20 bytes C++ > > > activemq-cppd.dll!activemq::transport::correlator::ResponseCorrelator::on= eway(const > > decaf::lang::Pointer > & command=3D{...}) Line 86 C++ > > > activemq-cppd.dll!activemq::core::ActiveMQConnection::oneway(decaf::lang:= :Pointer > command=3D{...}) Line 742 C++ > > > activemq-cppd.dll!activemq::core::ActiveMQSession::oneway(decaf::lang::Po= inter > command=3D{...}) Line 904 C++ > activemq-cppd.dll!activemq::core::ActiveMQConsumer::acknowledge() > Line 861 C++ > > activemq-cppd.dll!activemq::core::TransactionSynhcronization::beforeEnd() > Line 86 C++ > > activemq-cppd.dll!activemq::core::ActiveMQTransactionContext::beforeEnd() > Line 192 + 0x41 bytes C++ > > activemq-cppd.dll!activemq::core::ActiveMQTransactionContext::commit() > Line 130 C++ > activemq-cppd.dll!activemq::core::ActiveMQSession::commit() Line > 190 C++ > <<<<>>>>>> > activemq-cppd.dll!activemq::core::ActiveMQConsumer::dispatch(const > > decaf::lang::Pointer > & dispatch=3D{...}) Line 1021 C++ > > activemq-cppd.dll!activemq::core::ActiveMQSessionExecutor::dispatch(const > > decaf::lang::Pointer > & dispatch=3D{...}) Line 129 + 0x15 bytes C++ > > activemq-cppd.dll!activemq::core::ActiveMQSessionExecutor::iterate() > Line 167 C++ > activemq-cppd.dll!activemq::threads::DedicatedTaskRunner::run() > Line 111 + 0x19 bytes C++ > > > activemq-cppd.dll!decaf::lang::ThreadProperties::runCallback(decaf::lang:= :ThreadProperties > * properties=3D0x039cbd58) Line 135 + 0x11 bytes C++ > activemq-cppd.dll!`anonymous namespace'::threadWorker(void * > arg=3D0x039cbd58) Line 208 + 0x9 bytes C++ > msvcr100d.dll!_callthreadstartex() Line 314 + 0xf bytes C > msvcr100d.dll!_threadstartex(void * ptd=3D0x0396dba0) Line 297 C > kernel32.dll!775c33ca() > ntdll.dll!77dd9ed2() > ntdll.dll!77dd9ea5() > --=20 =D3scar Pernas Plaza. --0015174c3fb0523d5b049e7423d5--