activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "igor khaustov (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AMQCPP-376) Deadlock in IOTransport when network of brokers restart and failover is used.
Date Wed, 06 Jul 2011 06:24:16 GMT

    [ https://issues.apache.org/jira/browse/AMQCPP-376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13060319#comment-13060319
] 

igor khaustov commented on AMQCPP-376:
--------------------------------------

Additional backtrace:


+Thread 13+ (Thread 0x412f5940 (LWP 15374)):
*#0  0x00000032ef00a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0*
#1  0x0000000000dc5627 in decaf::internal::util::concurrent::ConditionImpl::wait (condition=0x10976bb8)
at decaf/internal/util/concurrent/unix/ConditionImpl.cpp:101
#2  0x0000000000bd9123 in decaf::util::concurrent::Mutex::wait (this=0x131cf9b8) at decaf/util/concurrent/Mutex.cpp:126
#3  0x0000000000ba8626 in decaf::lang::Thread::join (this=0x12b3a858) at decaf/lang/Thread.cpp:453
#4  0x0000000000d25f94 in activemq::threads::CompositeTaskRunner::shutdown (this=0x12fb33d8)
at activemq/threads/CompositeTaskRunner.cpp:77
#5  0x0000000000d260e6 in ~CompositeTaskRunner (this=0x12fb33d8) at activemq/threads/CompositeTaskRunner.cpp:44
#6  0x0000000000b20b93 in decaf::lang::Pointer<activemq::threads::CompositeTaskRunner,
decaf::util::concurrent::atomic::AtomicRefCounter>::onDeleteFunc (value=0x12fb33d8) at
./decaf/lang/Pointer.h:313
#7  0x0000000000b29a7e in ~Pointer (this=0x12b2e0d8) at ./decaf/lang/Pointer.h:145
#8  0x0000000000d58181 in ~InactivityMonitorData (this=0x12b2e018) at activemq/transport/inactivity/InactivityMonitor.cpp:54
#9  0x0000000000d55870 in ~InactivityMonitor (this=0x12bb2408) at activemq/transport/inactivity/InactivityMonitor.cpp:251
#10 0x0000000000a740d9 in decaf::lang::Pointer<activemq::transport::Transport, decaf::util::concurrent::atomic::AtomicRefCounter>::onDeleteFunc
(value=0x12bb2408) at ./decaf/lang/Pointer.h:313
#11 0x0000000000a7fdc8 in ~Pointer (this=0x12b30028) at ./decaf/lang/Pointer.h:145
#12 0x0000000000d34cad in ~TransportFilter (this=0x12b30018) at activemq/transport/TransportFilter.cpp:37
#13 0x0000000000d793cd in ~WireFormatNegotiator (this=0x12b30018) at ./activemq/wireformat/WireFormatNegotiator.h:45
#14 0x0000000000d78e48 in ~OpenWireFormatNegotiator (this=0x12b30018) at activemq/wireformat/openwire/OpenWireFormatNegotiator.cpp:56
#15 0x0000000000a740d9 in decaf::lang::Pointer<activemq::transport::Transport, decaf::util::concurrent::atomic::AtomicRefCounter>::onDeleteFunc
(value=0x12b30018) at ./decaf/lang/Pointer.h:313
#16 0x0000000000a7fdc8 in ~Pointer (this=0x412f4d40) at ./decaf/lang/Pointer.h:145
#17 0x0000000000a835e7 in decaf::lang::Pointer<activemq::transport::Transport, decaf::util::concurrent::atomic::AtomicRefCounter>::reset
(this=0x412f4de0, value=0x0) at ./decaf/lang/Pointer.h:158
*#18 0x0000000000d47b75 in activemq::transport::failover::CloseTransportsTask::iterate (this=0x12b740e8)
at activemq/transport/failover/CloseTransportsTask.cpp:83*
#19 0x0000000000d25981 in activemq::threads::CompositeTaskRunner::iterate (this=0x12e77d38)
at activemq/threads/CompositeTaskRunner.cpp:175
#20 0x0000000000d25bd4 in activemq::threads::CompositeTaskRunner::run (this=0x12e77d38) at
activemq/threads/CompositeTaskRunner.cpp:107
#21 0x0000000000baae37 in decaf::lang::ThreadProperties::runCallback (properties=0x131cf2b8)
at decaf/lang/Thread.cpp:137
#22 0x0000000000ba9156 in threadWorker (arg=0x131cf2b8) at decaf/lang/Thread.cpp:190
#23 0x00000032ef006367 in start_thread () from /lib64/libpthread.so.0
#24 0x00000032ee4d30ad in clone () from /lib64/libc.so.6



+Thread 7+ (Thread 0x48ec9940 (LWP 15384)):
*#0  0x00000032ef00ce74 in __lll_lock_wait () from /lib64/libpthread.so.0#1  0x00000032ef008874
in _L_lock_106 () from /lib64/libpthread.so.0*
#2  0x00000032ef0082e0 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x0000000000dc5b78 in decaf::internal::util::concurrent::MutexImpl::lock (handle=0x12b4bd38)
at decaf/internal/util/concurrent/unix/MutexImpl.cpp:77
#4  0x0000000000bd9182 in decaf::util::concurrent::Mutex::lock (this=0x12b74100) at decaf/util/concurrent/Mutex.cpp:111
#5  0x0000000000d520b1 in decaf::util::AbstractCollection<decaf::lang::Pointer<activemq::transport::Transport,
decaf::util::concurrent::atomic::AtomicRefCounter> >::lock (this=0x12b740f8) at ./decaf/util/AbstractCollection.h:331
#6  0x0000000000bd87b9 in decaf::util::concurrent::Lock::lock (this=0x48ec8bd0) at decaf/util/concurrent/Lock.cpp:54
#7  0x0000000000bd892a in Lock (this=0x48ec8bd0, object=0x12b74188, intiallyLocked=true) at
decaf/util/concurrent/Lock.cpp:32
*#8  0x0000000000d479f3 in activemq::transport::failover::CloseTransportsTask::add (this=0x12b740e8,
transport=@0x48ec8c30) at activemq/transport/failover/CloseTransportsTask.cpp:46*
#9  0x0000000000b1b818 in activemq::transport::failover::FailoverTransport::handleTransportFailure
(this=0x12abb498, error=@0x48ec8db0) at activemq/transport/failover/FailoverTransport.cpp:485
#10 0x0000000000b41ad6 in activemq::transport::failover::FailoverTransportListener::onException
(this=0x12c05798, ex=@0x48ec8db0) at activemq/transport/failover/FailoverTransportListener.cpp:76
#11 0x0000000000d3492f in activemq::transport::TransportFilter::fire (this=0x12b30018, ex=@0x48ec8db0)
at activemq/transport/TransportFilter.cpp:54
#12 0x0000000000d3495d in activemq::transport::TransportFilter::onException (this=0x12b30018,
ex=@0x48ec8db0) at activemq/transport/TransportFilter.cpp:46
#13 0x0000000000d3492f in activemq::transport::TransportFilter::fire (this=0x12bb2408, ex=@0x48ec8db0)
at activemq/transport/TransportFilter.cpp:54
#14 0x0000000000d3495d in activemq::transport::TransportFilter::onException (this=0x12bb2408,
ex=@0x48ec8db0) at activemq/transport/TransportFilter.cpp:46
#15 0x0000000000d5563c in activemq::transport::inactivity::InactivityMonitor::onException
(this=0x12bb2408, ex=@0x48ec8db0) at activemq/transport/inactivity/InactivityMonitor.cpp:312
#16 0x0000000000d57b0f in activemq::transport::inactivity::AsyncSignalReadErrorkTask::iterate
(this=0x10975158) at activemq/transport/inactivity/InactivityMonitor.cpp:158
#17 0x0000000000d25981 in 8activemq::threads::CompositeTaskRunner::iterate (this=0x12fb33d8)*
at activemq/threads/CompositeTaskRunner.cpp:175
#18 0x0000000000d25bd4 in activemq::threads::CompositeTaskRunner::run (this=0x12fb33d8) at
activemq/threads/CompositeTaskRunner.cpp:107
#19 0x0000000000baae37 in decaf::lang::ThreadProperties::runCallback (properties=0x131cf9b8)
at decaf/lang/Thread.cpp:137
#20 0x0000000000ba9156 in threadWorker (arg=0x131cf9b8) at decaf/lang/Thread.cpp:190
#21 0x00000032ef006367 in start_thread () from /lib64/libpthread.so.0
#22 0x00000032ee4d30ad in clone () from /lib64/libc.so.6


> Deadlock in IOTransport when network of brokers restart and failover is used. 
> ------------------------------------------------------------------------------
>
>                 Key: AMQCPP-376
>                 URL: https://issues.apache.org/jira/browse/AMQCPP-376
>             Project: ActiveMQ C++ Client
>          Issue Type: Bug
>          Components: Other C++ Clients
>    Affects Versions: 3.4.0
>         Environment: ActiveMQ-CPP  ver - 3.4.0
> Broker  5.3.1
> Machine: Linux mars 2.6.18-128.el5 #1 SMP Wed Dec 17 11:41:38 EST 2008 x86_64 x86_64
x86_64 GNU/Linux
> gcc version: 4.1.2 20080704 (Red Hat 4.1.2-44))
>            Reporter: igor khaustov
>            Assignee: Timothy Bish
>
> The problem description:
> We  run Network of brokers ( 4 in number ) . 
> Broker URI : broker URI 'failover://(tcp://10.10.13.20:61616,tcp://10.10.13.22:61616,tcp://10.10.13.24:61616,tcp://10.10.13.26:61616)?randomize=true&connection.closeTimeout=10000&transport.soTimeout=3000&timeout=3000&connection.useAsyncSend=true&connection.alwaysSyncSend=false'
> Producer loads broker with 1000 message/sec . We testing the producer behavior while
failover by  restarting all brokers in row ( all 4 ) while sending the messages and get deadlock
as shown below .
> Note: the problem tested only with network on brokers .
> The backtrace ( only relevant threads ):
> +Thread 16 (process 26892):+
> *#0  0x00000032ef00ce74 in __lll_lock_wait () from /lib64/libpthread.so.0*
> #1  0x00000032ef008874 in _L_lock_106 () from /lib64/libpthread.so.0
> #2  0x00000032ef0082e0 in pthread_mutex_lock () from /lib64/libpthread.so.0
> #3  0x0000000000dc5a04 in decaf::internal::util::concurrent::MutexImpl::lock (handle=0xfefdd38)
at decaf/internal/util/concurrent/unix/MutexImpl.cpp:77
> #4  0x0000000000bd9092 in decaf::util::concurrent::Mutex::lock (this=0xff54100) at decaf/util/concurrent/Mutex.cpp:111
> #5  0x0000000000d51f3f in decaf::util::AbstractCollection<decaf::lang::Pointer<activemq::transport::Transport,
decaf::util::concurrent::atomic::AtomicRefCounter> >::lock (this=0xff540f8) at ./decaf/util/AbstractCollection.h:331
> #6  0x0000000000bd86c9 in decaf::util::concurrent::Lock::lock (this=0x4c7b9c90) at decaf/util/concurrent/Lock.cpp:54
> #7  0x0000000000bd883a in Lock (this=0x4c7b9c90, object=0xff54188, intiallyLocked=true)
at decaf/util/concurrent/Lock.cpp:32
> *#8  0x0000000000d47a77 in activemq::transport::failover::CloseTransportsTask::add (this=0xff540e8,
transport=@0x4c7b9cf0) at activemq/transport/failover/CloseTransportsTask.cpp:46*
> #9  0x0000000000b1b748 in activemq::transport::failover::FailoverTransport::handleTransportFailure
(this=0xffed498, error=@0x4c7b9ee0) at activemq/transport/failover/FailoverTransport.cpp:483
> #10 0x0000000000b41a06 in activemq::transport::failover::FailoverTransportListener::onException
(this=0xfde2e58, ex=@0x4c7b9ee0) at activemq/transport/failover/FailoverTransportListener.cpp:76
> #11 0x0000000000d34813 in activemq::transport::TransportFilter::fire (this=0x10627498,
ex=@0x4c7b9ee0) at activemq/transport/TransportFilter.cpp:54
> #12 0x0000000000d34841 in activemq::transport::TransportFilter::onException (this=0x10627498,
ex=@0x4c7b9ee0) at activemq/transport/TransportFilter.cpp:46
> #13 0x0000000000d34813 in activemq::transport::TransportFilter::fire (this=0xfeeb558,
ex=@0x4c7b9ee0) at activemq/transport/TransportFilter.cpp:54
> #14 0x0000000000d34841 in activemq::transport::TransportFilter::onException (this=0xfeeb558,
ex=@0x4c7b9ee0) at activemq/transport/TransportFilter.cpp:46
> #15 0x0000000000d554c8 in activemq::transport::inactivity::InactivityMonitor::onException
(this=0xfeeb558, ex=@0x4c7b9ee0) at activemq/transport/inactivity/InactivityMonitor.cpp:312
> #16 0x0000000000d34813 in activemq::transport::TransportFilter::fire (this=0x1020c118,
ex=@0x4c7b9ee0) at activemq/transport/TransportFilter.cpp:54
> #17 0x0000000000d34841 in activemq::transport::TransportFilter::onException (this=0x1020c118,
ex=@0x4c7b9ee0) at activemq/transport/TransportFilter.cpp:46
> #18 0x0000000000d326f2 in activemq::transport::IOTransport::fire (this=0xdce10b8, ex=@0x4c7b9ee0)
at activemq/transport/IOTransport.cpp:87
> #19 0x0000000000d32982 in activemq::transport::IOTransport::run (this=0xdce10b8) at activemq/transport/IOTransport.cpp:264
> #20 0x0000000000baad49 in decaf::lang::ThreadProperties::runCallback (properties=0x105871d8)
at decaf/lang/Thread.cpp:137
> #21 0x0000000000ba9068 in threadWorker (arg=0x105871d8) at decaf/lang/Thread.cpp:190
> #22 0x00000032ef006367 in start_thread () from /lib64/libpthread.so.0
> #23 0x00000032ee4d30ad in clone () from /lib64/libc.so.6
> +Thread 9 (process 14470):+
> *#0  0x00000032ef00a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0*
> #1  0x0000000000dc54b3 in decaf::internal::util::concurrent::ConditionImpl::wait (condition=0x1072d2b8)
at decaf/internal/util/concurrent/unix/ConditionImpl.cpp:101
> #2  0x0000000000bd9033 in decaf::util::concurrent::Mutex::wait (this=0x105871d8) at decaf/util/concurrent/Mutex.cpp:126
> #3  0x0000000000ba8538 in decaf::lang::Thread::join (this=0x12a4a418) at decaf/lang/Thread.cpp:452
> #4  0x0000000000d32c28 in activemq::transport::IOTransport::close (this=0xdce10b8) at
activemq/transport/IOTransport.cpp:222
> #5  0x0000000000d34bfe in activemq::transport::TransportFilter::close (this=0x1020c118)
at activemq/transport/TransportFilter.cpp:106
> #6  0x0000000000b47d3a in activemq::transport::tcp::TcpTransport::close (this=0x1020c118)
at activemq/transport/tcp/TcpTransport.cpp:74
> #7  0x0000000000d34bfe in activemq::transport::TransportFilter::close (this=0xfeeb558)
at activemq/transport/TransportFilter.cpp:106
> #8  0x0000000000d554ec in activemq::transport::inactivity::InactivityMonitor::close (this=0xfeeb558)
at activemq/transport/inactivity/InactivityMonitor.cpp:300
> #9  0x0000000000d77867 in activemq::wireformat::openwire::OpenWireFormatNegotiator::close
(this=0x10627498) at activemq/wireformat/openwire/OpenWireFormatNegotiator.cpp:248
> *#10 0x0000000000d478ff in activemq::transport::failover::CloseTransportsTask::iterate
(this=0xff540e8) at activemq/transport/failover/CloseTransportsTask.cpp:75*
> #11 0x0000000000d25891 in activemq::threads::CompositeTaskRunner::iterate (this=0xddc0108)
at activemq/threads/CompositeTaskRunner.cpp:173
> #12 0x0000000000d25ae4 in activemq::threads::CompositeTaskRunner::run (this=0xddc0108)
at activemq/threads/CompositeTaskRunner.cpp:107
> #13 0x0000000000baad49 in decaf::lang::ThreadProperties::runCallback (properties=0xfeeb2b8)
at decaf/lang/Thread.cpp:137
> #14 0x0000000000ba9068 in threadWorker (arg=0xfeeb2b8) at decaf/lang/Thread.cpp:190
> #15 0x00000032ef006367 in start_thread () from /lib64/libpthread.so.0
> #16 0x00000032ee4d30ad in clone () from /lib64/libc.so.6
> As you can see +Thread 16+ is on lock_wait for *_synchronized( &transports )_* in
activemq::transport::failover::CloseTransportsTask::add .
> The *_synchronized( &transports )_* in locked by +Thread 9+ in activemq::threads::CompositeTaskRunner::iterate
. But  +Thread 9+ is on pthread_cond_wait which has to be signalled by the +Thread 16+.
> Kind regards .
> Igor.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message