activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eddie Fast (JIRA)" <j...@apache.org>
Subject [jira] Commented: (AMQCPP-277) Freeze when creating multiple Consumers
Date Fri, 05 Mar 2010 02:39:44 GMT

    [ https://issues.apache.org/activemq/browse/AMQCPP-277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=58050#action_58050
] 

Eddie Fast commented on AMQCPP-277:
-----------------------------------


Just an update on this issue.  We've upgraded to 3.1.1 and still have this deadlock issue.

I've been able to reproduce on both the example main.cpp, as well as the new unit test that
was added, testCreateManyConsumersAndSetListeners().

In the unit test, I was getting deadlocks about 90% of the time in the 10 or so tests I ran,
and it seemed to deadlock after around 70-150 consumers had been created.

This was in the ReleaseDLL config.  The only change I made was to increase the number of consumers
to 500 just because the deadlocks were happening around 100 or so.  Also, I'm using the .dll
versions of APR, so I made that change to the project.


I guess this could be a problem of how I'm compiling APR and/or Activemq.  That's always sitting
in the back of my mind.  I'm assuming I did it the "standard" way.  But like I said, we don't
have this problem when we revert back to 2.2.2, and that was compiled the same way.


I'm going to attach callstacks for all threads from a deadlock in the unit test.   Let me
know what else I can provide to help narrow this down.


In this case, it deadlocked with ix=164.  The callstacks look the same regardless of how many
it has created.


List of threads:

0	>	4456	Main Thread	Main Thread	decaf::internal::util::concurrent::ConditionImpl::wait
Normal	0
0	 	1804	Worker Thread	`anonymous namespace'::threadWorker	decaf::internal::util::concurrent::ConditionImpl::wait
Normal	0
0	 	5704	Worker Thread	`anonymous namespace'::threadWorker	decaf::internal::util::concurrent::ConditionImpl::wait
Normal	0
0	 	7408	Worker Thread	`anonymous namespace'::threadWorker	decaf::internal::util::concurrent::ConditionImpl::wait
Normal	0
0	 	3312	Worker Thread	`anonymous namespace'::threadWorker	decaf::internal::util::concurrent::ConditionImpl::wait
Normal	0


---------------


4456	Main Thread

 	ntdll.dll!771df871() 	
 	[Frames below may be incorrect and/or missing, no symbols loaded for ntdll.dll]	
 	ntdll.dll!771df871() 	
 	KernelBase.dll!763c0816() 	
 	kernel32.dll!74fb1138() 	
 	activemq-cpp.dll!decaf::internal::util::concurrent::ConditionImpl::wait(decaf::util::concurrent::ConditionHandle
* condition=0x00bc23a0, __int64 mills=4294967295, __int64 nanos=0)  Line 110 + 0x10 bytes
C++
 	activemq-cpp.dll!decaf::util::concurrent::Mutex::wait(__int64 millisecs=4294967295, int
nanos=0)  Line 124 + 0x20 bytes	C++
 	activemq-cpp.dll!decaf::lang::Thread::join(__int64 millisecs=4294967295, unsigned int nanos=0)
 Line 464 + 0x36 bytes	C++
 	activemq-cpp.dll!decaf::lang::Thread::join()  Line 421	C++
 	activemq-cpp.dll!activemq::threads::DedicatedTaskRunner::shutdown()  Line 83	C++
 	activemq-cpp.dll!activemq::core::ActiveMQSessionExecutor::stop()  Line 110	C++
 	activemq-cpp.dll!activemq::core::ActiveMQSession::stop()  Line 807	C++
 	activemq-cpp.dll!activemq::core::ActiveMQConsumer::setMessageListener(cms::MessageListener
* listener=0x0018fadc)  Line 523	C++
>	vs2005-activemq-unittests.exe!activemq::core::ActiveMQSessionTest::testCreateManyConsumersAndSetListeners()
 Line 127	C++
 	vs2005-activemq-unittests.exe!CppUnit::TestCaller<activemq::core::ActiveMQSessionTest>::runTest()
 Line 173	C++
 	vs2005-activemq-unittests.exe!CppUnit::TestCaseMethodFunctor::operator()()  Line 33	C++
 	vs2005-activemq-unittests.exe!CppUnit::DefaultProtector::protect(const CppUnit::Functor
& functor={...}, const CppUnit::ProtectorContext & context={...})  Line 15 + 0x11
bytes	C++
 	vs2005-activemq-unittests.exe!CppUnit::ProtectorChain::ProtectFunctor::operator()()  Line
20 + 0x13 bytes	C++
 	vs2005-activemq-unittests.exe!CppUnit::ProtectorChain::protect(const CppUnit::Functor &
functor={...}, const CppUnit::ProtectorContext & context={...})  Line 79	C++
 	vs2005-activemq-unittests.exe!CppUnit::TestResult::protect(const CppUnit::Functor &
functor={...}, CppUnit::Test * test=0x00b1ba08, const std::basic_string<char,std::char_traits<char>,std::allocator<char>
> & shortDescription="")  Line 178 + 0x18 bytes	C++
 	vs2005-activemq-unittests.exe!CppUnit::TestCase::run(CppUnit::TestResult * result=0x00b3a770)
 Line 92 + 0x4a bytes	C++
 	vs2005-activemq-unittests.exe!CppUnit::TestComposite::doRunChildTests(CppUnit::TestResult
* controller=0x00b3a770)  Line 64 + 0x14 bytes	C++
 	vs2005-activemq-unittests.exe!CppUnit::TestComposite::run(CppUnit::TestResult * result=0x00b3a770)
 Line 24	C++
 	vs2005-activemq-unittests.exe!CppUnit::TestComposite::doRunChildTests(CppUnit::TestResult
* controller=0x00b3a770)  Line 64 + 0x14 bytes	C++
 	vs2005-activemq-unittests.exe!CppUnit::TestComposite::run(CppUnit::TestResult * result=0x00b3a770)
 Line 24	C++
 	vs2005-activemq-unittests.exe!CppUnit::TestResult::runTest(CppUnit::Test * test=0x00b39390)
 Line 146	C++
 	vs2005-activemq-unittests.exe!CppUnit::TestRunner::run(CppUnit::TestResult & controller={...},
const std::basic_string<char,std::char_traits<char>,std::allocator<char> >
& testPath="")  Line 97	C++
 	vs2005-activemq-unittests.exe!CppUnit::TextTestRunner::run(std::basic_string<char,std::char_traits<char>,std::allocator<char>
> testName="", bool doWait=false, bool doPrintResult=true, bool doPrintProgress=true) 
Line 66	C++
 	vs2005-activemq-unittests.exe!main(int argc=1, char * * argv=0x00c38050)  Line 111 + 0x24
bytes	C++
 	vs2005-activemq-unittests.exe!__tmainCRTStartup()  Line 586 + 0x17 bytes	C
 	kernel32.dll!74fb3677() 	
 	ntdll.dll!771f9d72() 	
 	ntdll.dll!771f9d45() 	
 	vs2005-activemq-unittests.exe!___sse2_available_init()  + 0x3cff2 bytes	C++
 	fffb5c8d()	


---------------


1804	Worker Thread

 	ntdll.dll!771df871() 	
 	[Frames below may be incorrect and/or missing, no symbols loaded for ntdll.dll]	
 	ntdll.dll!771df871() 	
 	KernelBase.dll!763c0816() 	
 	kernel32.dll!74fb1138() 	
>	activemq-cpp.dll!decaf::internal::util::concurrent::ConditionImpl::wait(decaf::util::concurrent::ConditionHandle
* condition=0x00b408c8, __int64 mills=4294967295, __int64 nanos=0)  Line 110 + 0x10 bytes
C++
 	activemq-cpp.dll!decaf::internal::util::concurrent::ConditionImpl::wait(decaf::util::concurrent::ConditionHandle
* condition=0x00b408c8)  Line 71 + 0x11 bytes	C++
 	activemq-cpp.dll!decaf::util::concurrent::Mutex::wait()  Line 95 + 0xf bytes	C++
 	activemq-cpp.dll!decaf::util::StlQueue<std::pair<decaf::lang::Runnable *,decaf::util::concurrent::TaskListener
*> >::wait()  Line 269	C++
 	activemq-cpp.dll!decaf::util::concurrent::ThreadPool::deQueueTask()  Line 151	C++
 	activemq-cpp.dll!decaf::util::concurrent::PooledThread::run()  Line 68 + 0x1d bytes	C++
 	activemq-cpp.dll!decaf::lang::ThreadProperties::runCallback(decaf::lang::ThreadProperties
* properties=0x00b40c28)  Line 133 + 0x11 bytes	C++
 	activemq-cpp.dll!`anonymous namespace'::threadWorker(void * arg=0x00b40c28)  Line 204 +
0x9 bytes	C++
 	msvcr90.dll!727d3433() 	
 	msvcr90.dll!727d34c7() 	
 	kernel32.dll!74fb3677() 	
 	ntdll.dll!771f9d72() 	
 	ntdll.dll!771f9d45() 	


---------------


5704	Worker Thread

 	ntdll.dll!771df871() 	
 	[Frames below may be incorrect and/or missing, no symbols loaded for ntdll.dll]	
 	ntdll.dll!771df871() 	
 	KernelBase.dll!763c0816() 	
 	kernel32.dll!74fb1138() 	
>	activemq-cpp.dll!decaf::internal::util::concurrent::ConditionImpl::wait(decaf::util::concurrent::ConditionHandle
* condition=0x00b408c8, __int64 mills=4294967295, __int64 nanos=0)  Line 110 + 0x10 bytes
C++
 	activemq-cpp.dll!decaf::internal::util::concurrent::ConditionImpl::wait(decaf::util::concurrent::ConditionHandle
* condition=0x00b408c8)  Line 71 + 0x11 bytes	C++
 	activemq-cpp.dll!decaf::util::concurrent::Mutex::wait()  Line 95 + 0xf bytes	C++
 	activemq-cpp.dll!decaf::util::StlQueue<std::pair<decaf::lang::Runnable *,decaf::util::concurrent::TaskListener
*> >::wait()  Line 269	C++
 	activemq-cpp.dll!decaf::util::concurrent::ThreadPool::deQueueTask()  Line 151	C++
 	activemq-cpp.dll!decaf::util::concurrent::PooledThread::run()  Line 68 + 0x1d bytes	C++
 	activemq-cpp.dll!decaf::lang::ThreadProperties::runCallback(decaf::lang::ThreadProperties
* properties=0x00b40fa8)  Line 133 + 0x11 bytes	C++
 	activemq-cpp.dll!`anonymous namespace'::threadWorker(void * arg=0x00b40fa8)  Line 204 +
0x9 bytes	C++
 	msvcr90.dll!727d3433() 	
 	msvcr90.dll!727d34c7() 	
 	kernel32.dll!74fb3677() 	
 	ntdll.dll!771f9d72() 	
 	ntdll.dll!771f9d45() 	


---------------


7408	Worker Thread


 	ntdll.dll!771df871() 	
 	[Frames below may be incorrect and/or missing, no symbols loaded for ntdll.dll]	
 	ntdll.dll!771df871() 	
 	KernelBase.dll!763c0816() 	
 	msctf.dll!76cf1651() 	
 	kernel32.dll!74fb1138() 	
>	activemq-cpp.dll!decaf::internal::util::concurrent::ConditionImpl::wait(decaf::util::concurrent::ConditionHandle
* condition=0x00b408c8, __int64 mills=4294967295, __int64 nanos=0)  Line 110 + 0x10 bytes
C++
 	activemq-cpp.dll!decaf::internal::util::concurrent::ConditionImpl::wait(decaf::util::concurrent::ConditionHandle
* condition=0x00b408c8)  Line 71 + 0x11 bytes	C++
 	activemq-cpp.dll!decaf::util::concurrent::Mutex::wait()  Line 95 + 0xf bytes	C++
 	activemq-cpp.dll!decaf::util::StlQueue<std::pair<decaf::lang::Runnable *,decaf::util::concurrent::TaskListener
*> >::wait()  Line 269	C++
 	activemq-cpp.dll!decaf::util::concurrent::ThreadPool::deQueueTask()  Line 151	C++
 	activemq-cpp.dll!decaf::util::concurrent::PooledThread::run()  Line 68 + 0x1d bytes	C++
 	activemq-cpp.dll!decaf::lang::ThreadProperties::runCallback(decaf::lang::ThreadProperties
* properties=0x00b41408)  Line 133 + 0x11 bytes	C++
 	activemq-cpp.dll!`anonymous namespace'::threadWorker(void * arg=0x00b41408)  Line 204 +
0x9 bytes	C++
 	msvcr90.dll!727d3433() 	
 	msvcr90.dll!727d34c7() 	
 	kernel32.dll!74fb3677() 	
 	ntdll.dll!771f9d72() 	
 	ntdll.dll!771f9d45() 	


---------------


3312	Worker Thread


 	ntdll.dll!771df871() 	
 	[Frames below may be incorrect and/or missing, no symbols loaded for ntdll.dll]	
 	ntdll.dll!771df871() 	
 	KernelBase.dll!763c0816() 	
 	kernel32.dll!74fb1138() 	
>	activemq-cpp.dll!decaf::internal::util::concurrent::ConditionImpl::wait(decaf::util::concurrent::ConditionHandle
* condition=0x00b485b0, __int64 mills=4294967295, __int64 nanos=0)  Line 110 + 0x10 bytes
C++
 	activemq-cpp.dll!decaf::internal::util::concurrent::ConditionImpl::wait(decaf::util::concurrent::ConditionHandle
* condition=0x00b485b0)  Line 71 + 0x11 bytes	C++
 	activemq-cpp.dll!decaf::util::concurrent::Mutex::wait()  Line 95 + 0xf bytes	C++
 	activemq-cpp.dll!decaf::util::StlQueue<decaf::lang::Pointer<activemq::commands::Command,decaf::lang::AtomicRefCounter>
>::wait()  Line 269	C++
 	activemq-cpp.dll!activemq::transport::mock::InternalCommandListener::run()  Line 75	C++
 	activemq-cpp.dll!decaf::lang::ThreadProperties::runCallback(decaf::lang::ThreadProperties
* properties=0x00b43050)  Line 133 + 0x11 bytes	C++
 	activemq-cpp.dll!`anonymous namespace'::threadWorker(void * arg=0x00b43050)  Line 204 +
0x9 bytes	C++
 	msvcr90.dll!727d3433() 	
 	msvcr90.dll!727d34c7() 	
 	kernel32.dll!74fb3677() 	
 	ntdll.dll!771f9d72() 	
 	ntdll.dll!771f9d45() 	


> Freeze when creating multiple Consumers
> ---------------------------------------
>
>                 Key: AMQCPP-277
>                 URL: https://issues.apache.org/activemq/browse/AMQCPP-277
>             Project: ActiveMQ C++ Client
>          Issue Type: Bug
>    Affects Versions: 3.1
>         Environment: Windows 7 x64, ActiveMQ Server 5.3, Java 6.0.17, Visual Studio 2008
SP1, Windows 7 SDK, APR 1.3.9
>            Reporter: Eddie Fast
>            Assignee: Timothy Bish
>             Fix For: 3.2.0
>
>
> We create multiple MessageConsumers with different selectors.  This can be upwards of
50 or so consumers per client.
> We recently upgraded to 3.1 from 2.2.2.    We also upgraded our ActiveMQ Server to 5.3
from 5.2.
> After upgrading, we've been seeing random, but frequent freezing in our clients.  I've
narrowed it down to where it creates the MessageConsumer and calls setMessageListener();
> It seems to be stuck, waiting for a thread to join.
> I've modified the main.cpp to exhibit this behavior, but I've only been able to reproduce
it randomly.   It seems to happen more frequently if you stick a breakpoint on this line in
Thread.cpp,  Thread::join():
> Thread::join( INFINITE, 0 );
> Here's the callstack where it freezes, which is consistent in our clients as well as
the main.cpp example:
>  	ntdll.dll!76fff871() 	
>  	[Frames below may be incorrect and/or missing, no symbols loaded for ntdll.dll]	
>  	ntdll.dll!76fff871() 	
>  	KernelBase.dll!75e20816() 	
>  	kernel32.dll!755b1138() 	
>  	activemq-cppd.dll!decaf::internal::util::concurrent::ConditionImpl::wait(decaf::util::concurrent::ConditionHandle
* condition=0x0239e0e0, __int64 mills=4294967295, __int64 nanos=0)  Line 110 + 0x10 bytes
C++
>  	activemq-cppd.dll!decaf::util::concurrent::Mutex::wait(__int64 millisecs=4294967295,
int nanos=0)  Line 124 + 0x20 bytes	C++
>  	activemq-cppd.dll!decaf::lang::Thread::join(__int64 millisecs=4294967295, unsigned
int nanos=0)  Line 464 + 0x36 bytes	C++
>  	activemq-cppd.dll!decaf::lang::Thread::join()  Line 421	C++
> >	activemq-cppd.dll!activemq::threads::DedicatedTaskRunner::shutdown()  Line 83	C++
>  	activemq-cppd.dll!activemq::core::ActiveMQSessionExecutor::stop()  Line 110	C++
>  	activemq-cppd.dll!activemq::core::ActiveMQSession::stop()  Line 807	C++
>  	activemq-cppd.dll!activemq::core::ActiveMQConsumer::setMessageListener(cms::MessageListener
* listener=0x0018fe0c)  Line 523	C++
>  	vs2005-activemq-example.exe!HelloWorldConsumer::run()  Line 247 + 0x5b bytes	C++
>  	activemq-cppd.dll!decaf::lang::ThreadProperties::runCallback(decaf::lang::ThreadProperties
* properties=0x01df7f68)  Line 133 + 0x11 bytes	C++
>  	activemq-cppd.dll!`anonymous namespace'::threadWorker(void * arg=0x01df7f68)  Line
204 + 0x9 bytes	C++
>  	msvcr90d.dll!_callthreadstartex()  Line 348 + 0xf bytes	C
>  	msvcr90d.dll!_threadstartex(void * ptd=0x01dfafd8)  Line 331	C
> To modify main.cpp, it's pretty simple.  Instead of creating 1 MessageConsumer in HelloWorldConsumer::run(),
I've created 50:
> {quote}
> // put this line with the class variables
> std::vector<MessageConsumer*> consumers;
> for ( int i = 0; i < 50; i++ )
> {{
>          // Create a MessageConsumer from the Session to the Topic or Queue
>          consumers.push_back( session->createConsumer( destination ) );
>          consumers.back()->setMessageListener( this );
> }}
> {quote}
> I'm using the standard .conf for the server.    I've reverted to 2.2.2 and have no issues.
> Any ideas?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message