activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Timothy Bish <tabish...@gmail.com>
Subject Re: ActiveMq-Cpp deadlock recovery
Date Tue, 27 Oct 2009 14:45:06 GMT
On Tue, 2009-10-27 at 07:35 -0700, bpmuser wrote:
> I'm having a bit of trouble with a deadlock in my ActiveMq-Cpp dll.  Every
> time my producer tries to send a message the application completely locks
> up.  My program only has a producer defined with no read expected.
> 
> At some point over night the application locked.  I killed the process and
> restarted the application.  Now it constantly freezes on every send.  This
> has happened before and a reset will clean things up but I would rather try
> and resolve the issue to avoid the reboot. The call stacks below shows 1
> thread performing a read operation, while two other threads are blocked on a
> wait condition, and several other threads blocked as well.  Everything seems
> to be waiting on kernel32.dll!_WaitForSingleObject@8(), but I'm not sure
> where the lock is being held.
> 
> My program has two threads created explicitly (1 main + 1 worker), but there
> are about 11 threads running in the program(?).
> 
> Does anyone know what might be causing this problem?  Is there a way to
> clear this deadlock without rebooting?  Any suggestions would be really
> appreciated.

Most likely cause is that you have producer flow control enabled on the
broker and the client is being blocked because there is no more room on
the broker for new messages.

See: http://activemq.apache.org/my-producer-blocks.html

Regards
Tim.


> 
> Thanks,
> Brian!
> 
> Thread 1:
>  	ntdll.dll!_KiFastSystemCallRet@0() 	
>  	ntdll.dll!_NtWaitForSingleObject@12()  + 0xc bytes	
>  	mswsock.dll!_SockWaitForSingleObject@16()  + 0x419 bytes	
>  	mswsock.dll!_WSPRecv@36()  + 0xadf bytes	
>  	ws2_32.dll!_WSARecv@28()  + 0x6d bytes	
>  	libapr-1.dll!6eecb199() 	
>  	[Frames below may be incorrect and/or missing, no symbols loaded for
> libapr-1.dll]	
> >	activemq-cppd.dll!decaf::net::SocketInputStream::read(unsigned char *
> buffer=0x02cfbd58, unsigned int offset=0, unsigned int bufferSize=1)  Line
> 179 + 0x1b bytes	C++
>  	activemq-cppd.dll!decaf::io::BufferedInputStream::bufferData()  Line 267 +
> 0x37 bytes	C++
>  	activemq-cppd.dll!decaf::io::BufferedInputStream::read(unsigned char *
> targetBuffer=0x02cfc770, unsigned int offset=0, unsigned int
> targetBufferSize=4)  Line 188 + 0xb bytes	C++
>  	activemq-cppd.dll!decaf::io::DataInputStream::readAllData(unsigned char *
> buffer=0x02cfc770, unsigned int length=4)  Line 375 + 0x23 bytes	C++
>  	activemq-cppd.dll!decaf::io::DataInputStream::readInt()  Line 166	C++
>  
> activemq-cppd.dll!activemq::wireformat::openwire::OpenWireFormat::unmarshal(const
> activemq::transport::Transport * transport=0x02cfb7e0,
> decaf::io::DataInputStream * dis=0x02cfc758)  Line 267	C++
>  	activemq-cppd.dll!activemq::transport::IOTransport::run()  Line 233 + 0x4b
> bytes	C++
>  
> activemq-cppd.dll!decaf::lang::ThreadProperties::runCallback(decaf::lang::ThreadProperties
> * properties=0x02cf9788)  Line 131 + 0x11 bytes	C++
>  	activemq-cppd.dll!`anonymous namespace'::threadWorker(void *
> arg=0x02cf9788)  Line 197 + 0x9 bytes	C++
>  	msvcr90d.dll!_callthreadstartex()  Line 348 + 0xf bytes	C
>  	msvcr90d.dll!_threadstartex(void * ptd=0x02cfd228)  Line 331	C
>  	kernel32.dll!@BaseThreadInitThunk@12()  + 0x12 bytes	
>  	ntdll.dll!___RtlUserThreadStart@8()  + 0x27 bytes	
>  	ntdll.dll!__RtlUserThreadStart@8()  + 0x1b bytes	
> 
> Thread 2:
>  	ntdll.dll!_KiFastSystemCallRet@0() 	
>  	ntdll.dll!_NtWaitForSingleObject@12()  + 0xc bytes	
>  	kernel32.dll!_WaitForSingleObjectEx@12()  + 0x84 bytes	
>  	kernel32.dll!_WaitForSingleObject@8()  + 0x12 bytes	
> >
> activemq-cppd.dll!decaf::internal::util::concurrent::ConditionImpl::wait(decaf::util::concurrent::ConditionHandle
> * condition=0x02cf7c10, __int64 mills=4294967295, __int64 nanos=0)  Line 105
> + 0x10 bytes	C++
>  
> activemq-cppd.dll!decaf::internal::util::concurrent::ConditionImpl::wait(decaf::util::concurrent::ConditionHandle
> * condition=0x02cf7c10)  Line 66 + 0x11 bytes	C++
>  	activemq-cppd.dll!decaf::util::concurrent::Mutex::wait()  Line 91 + 0x1b
> bytes	C++
>  	activemq-cppd.dll!activemq::threads::CompositeTaskRunner::run()  Line 118
> C++
>  
> activemq-cppd.dll!decaf::lang::ThreadProperties::runCallback(decaf::lang::ThreadProperties
> * properties=0x02cf7d08)  Line 131 + 0x11 bytes	C++
>  	activemq-cppd.dll!`anonymous namespace'::threadWorker(void *
> arg=0x02cf7d08)  Line 197 + 0x9 bytes	C++
>  	msvcr90d.dll!_callthreadstartex()  Line 348 + 0xf bytes	C
>  	msvcr90d.dll!_threadstartex(void * ptd=0x02cf81d0)  Line 331	C
>  	kernel32.dll!@BaseThreadInitThunk@12()  + 0x12 bytes	
>  	ntdll.dll!___RtlUserThreadStart@8()  + 0x27 bytes	
>  	ntdll.dll!__RtlUserThreadStart@8()  + 0x1b bytes	
> 
> Thread 3:
>  	ntdll.dll!_KiFastSystemCallRet@0() 	
>  	ntdll.dll!_NtWaitForSingleObject@12()  + 0xc bytes	
>  	kernel32.dll!_WaitForSingleObjectEx@12()  + 0x84 bytes	
>  	kernel32.dll!_WaitForSingleObject@8()  + 0x12 bytes	
> >
> activemq-cppd.dll!decaf::internal::util::concurrent::ConditionImpl::wait(decaf::util::concurrent::ConditionHandle
> * condition=0x02d01090, __int64 mills=4294967295, __int64 nanos=0)  Line 105
> + 0x10 bytes	C++
>  
> activemq-cppd.dll!decaf::internal::util::concurrent::ConditionImpl::wait(decaf::util::concurrent::ConditionHandle
> * condition=0x02d01090)  Line 66 + 0x11 bytes	C++
>  	activemq-cppd.dll!decaf::util::concurrent::Mutex::wait()  Line 91 + 0x1b
> bytes	C++
>  	activemq-cppd.dll!decaf::util::concurrent::CountDownLatch::await()  Line
> 53 + 0x17 bytes	C++
>  
> activemq-cppd.dll!activemq::transport::correlator::FutureResponse::getResponse() 
> Line 63	C++
>  
> activemq-cppd.dll!activemq::transport::correlator::ResponseCorrelator::request(const
> decaf::lang::Pointer<activemq::commands::Command,decaf::lang::AtomicRefCounter>
> & command={...})  Line 92 + 0x2d bytes	C++
>  
> activemq-cppd.dll!activemq::core::ActiveMQConnection::syncRequest(decaf::lang::Pointer<activemq::commands::Command,decaf::lang::AtomicRefCounter>
> command={...}, unsigned int timeout=0)  Line 617 + 0x3e bytes	C++
>  	activemq-cppd.dll!activemq::core::ActiveMQSession::send(cms::Message *
> message=0x02cf2da0, activemq::core::ActiveMQProducer * producer=0x02cff790,
> activemq::util::Usage * usage=0x00000000)  Line 716	C++
>  	activemq-cppd.dll!activemq::core::ActiveMQProducer::send(const
> cms::Destination * destination=0x02cff358, cms::Message *
> message=0x02cf2da0, int deliveryMode=0, int priority=4, __int64
> timeToLive=0)  Line 183	C++
>  	activemq-cppd.dll!activemq::core::ActiveMQProducer::send(const
> cms::Destination * destination=0x02cff358, cms::Message *
> message=0x02cf2da0)  Line 142	C++
>  	activemq-cppd.dll!activemq::core::ActiveMQProducer::send(cms::Message *
> message=0x02cf2da0)  Line 107	C++
>  	MFCLaserTrak2.exe!SharedActiveMqOutput::sendLaserSpot(int id=0, int
> time=334335617, float x=2840.4448, float y=810.95697, bool found=true)  Line
> 34 + 0x47 bytes	C++
>  	MFCLaserTrak2.exe!LaserSpotOutput::processImage(const unsigned char *
> img=0x054a0020)  Line 32 + 0x41 bytes	C++
>  	MFCLaserTrak2.exe!CaptureThread::captureNextFrame()  Line 136 + 0x21 bytes
> C++
>  	MFCLaserTrak2.exe!CaptureThreadExec::operator()()  Line 39 + 0xa bytes	C++
>  
> MFCLaserTrak2.exe!tbb::internal::thread_closure_0<CaptureThreadExec>::start_routine(void
> * c=0x7ff87ff8)  Line 76 + 0x8 bytes	C++
>  	msvcr90d.dll!_callthreadstartex()  Line 348 + 0xf bytes	C
>  	msvcr90d.dll!_threadstartex(void * ptd=0x02cf2780)  Line 331	C
>  	kernel32.dll!@BaseThreadInitThunk@12()  + 0x12 bytes	
>  	ntdll.dll!___RtlUserThreadStart@8()  + 0x27 bytes	
>  	ntdll.dll!__RtlUserThreadStart@8()  + 0x1b bytes	
> 
> Remaining threads:
> >	ntdll.dll!_KiFastSystemCallRet@0() 	
>  	ntdll.dll!_NtWaitForSingleObject@12()  + 0xc bytes	
>  	kernel32.dll!_WaitForSingleObjectEx@12()  + 0x84 bytes	
>  	kernel32.dll!_WaitForSingleObject@8()  + 0x12 bytes	
>  	tbb_debug.dll!1001734f() 	
>  	[Frames below may be incorrect and/or missing, no symbols loaded for
> tbb_debug.dll]	
>  	tbb_debug.dll!1001050c() 	
>  	tbb_debug.dll!1001dc2b() 	
>  	tbb_debug.dll!1001328e() 	
> 
-- 
Tim Bish
http://fusesource.com
http://timbish.blogspot.com/




Mime
View raw message