activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From skate056 <skate...@gmail.com>
Subject Re: ActiveMQ message dequeuing hangs
Date Mon, 02 Jun 2014 15:52:48 GMT
Hello,

We were able to reproduce this issue with just ActiveMQ (minus our
application). It seems that when the consumers throw a exceptions
frequently, something goes wrong trying to rollback the transaction and the
LevelDB. It seems that a LevelDB thread is locking it up.

Attached is a test which simulates the behavior. For us it failed after
50000 messages but the number may vary on the machine.
We suspect the following thread to be the culprit as it continuously seems
to be blocked.

Thread 43139: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=20,
line=186 (Compiled frame [deoptimized])
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt()
@bci=7, line=834 (Compiled frame [deoptimized])
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(int)
@bci=89, line=994 (Compiled frame [deoptimized])
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(int)
@bci=47, line=1303 (Interpreted frame)
 - java.util.concurrent.CountDownLatch.await() @bci=11, line=236
(Interpreted frame)
 - *org.apache.activemq.leveldb.LevelDBStore.rollback*(org.apache.activemq.command.TransactionId)
@bci=152, line=498 (Interpreted frame)
 - *org.apache.activemq.transaction.LocalTransaction.rollback*() @bci=111,
line=94 (Interpreted frame)
 -
org.apache.activemq.broker.TransactionBroker.rollbackTransaction(org.apache.activemq.broker.ConnectionContext,
org.apache.activemq.command.TransactionId) @bci=15, line=258 (Interpreted
frame)
 -
org.apache.activemq.broker.BrokerFilter.rollbackTransaction(org.apache.activemq.broker.ConnectionContext,
org.apache.activemq.command.TransactionId) @bci=12, line=142 (Interpreted
frame)
 -
org.apache.activemq.broker.MutableBrokerFilter.rollbackTransaction(org.apache.activemq.broker.ConnectionContext,
org.apache.activemq.command.TransactionId) @bci=12, line=147 (Interpreted
frame)
 -
org.apache.activemq.broker.TransportConnection.processRollbackTransaction(org.apache.activemq.command.TransactionInfo)
@bci=44, line=442 (Interpreted frame)
 -
org.apache.activemq.command.TransactionInfo.visit(org.apache.activemq.state.CommandVisitor)
@bci=146, line=104 (Compiled frame [deoptimized])
 -
org.apache.activemq.broker.TransportConnection.service(org.apache.activemq.command.Command)
@bci=52, line=292 (Compiled frame [deoptimized])
 -
org.apache.activemq.broker.TransportConnection$1.onCommand(java.lang.Object)
@bci=95, line=149 (Compiled frame [deoptimized])
 - org.apache.activemq.transport.MutexTransport.onCommand(java.lang.Object)
@bci=58, line=50 (Compiled frame [deoptimized])
 -
org.apache.activemq.transport.WireFormatNegotiator.onCommand(java.lang.Object)
@bci=35, line=113 (Compiled frame [deoptimized])
 -
org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(java.lang.Object)
@bci=162, line=270 (Compiled frame [deoptimized])
 -
org.apache.activemq.transport.TransportSupport.doConsume(java.lang.Object)
@bci=27, line=83 (Compiled frame [deoptimized])
 - org.apache.activemq.transport.tcp.TcpTransport.doRun() @bci=13, line=214
(Compiled frame [deoptimized])
 - org.apache.activemq.transport.tcp.TcpTransport.run() @bci=70, line=196
(Compiled frame [deoptimized])
 - java.lang.Thread.run() @bci=17, line=724 (Interpreted frame)

Thanks.
Saket


On Thu, May 29, 2014 at 5:03 PM, Saket Jha <skate056@gmail.com> wrote:

> I looked at the thread dump but it does not point to something obvious.
> Attached are the thread dumps.
>
> I am trying to simulate this on a pure vanilla ActiveMQ instance with some
> random data.
>
>
> On Tue, May 27, 2014 at 5:08 PM, ceposta [via ActiveMQ] <
> ml-node+s2283324n4681454h27@n4.nabble.com> wrote:
>
>> Take thread dumps while you attach consumers to see what's going on.
>> Any way to recreate it?
>>
>>
>> On Fri, May 23, 2014 at 9:07 AM, skate056 <[hidden email]
>> <http://user/SendEmail.jtp?type=node&node=4681454&i=0>> wrote:
>>
>> > Hello all,
>> >
>> > We have about 145517 messages on the queue but after dequeueing about
>> 1510
>> > messages the dequeueing  stops. ActiveMQ is using about 28% of memory
>> and
>> > the web ui responds. But the messages are not getting dequeued.
>> >
>> > After restarting ActiveMQ it resumes dequeueing but then again
>> processes
>> > about ~1500 messages and then hangs again.
>> >
>> > We are running 5.9.0. Any suggestions what the issue could be?
>> >
>> > Thanks,
>> > Saket
>> >
>> >
>> >
>> > --
>> > View this message in context:
>> >
>> http://activemq.2283324.n4.nabble.com/ActiveMQ-message-dequeuing-hangs-tp4681366.html
>> > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>> >
>>
>>
>>
>> --
>> *Christian Posta*
>> http://www.christianposta.com/blog
>> twitter: @christianposta
>>  http://www.christianposta.com/blog
>>
>>
>> ------------------------------
>>  If you reply to this email, your message will be added to the
>> discussion below:
>>
>> http://activemq.2283324.n4.nabble.com/ActiveMQ-message-dequeuing-hangs-tp4681366p4681454.html
>>  To unsubscribe from ActiveMQ message dequeuing hangs, click here
>> <http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4681366&code=c2thdGUwNTZAZ21haWwuY29tfDQ2ODEzNjZ8MzM0NzIxODA=>
>> .
>> NAML
>> <http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>
>
>


activeMqLoadTest.tar.gz (24K) <http://activemq.2283324.n4.nabble.com/attachment/4681579/0/activeMqLoadTest.tar.gz>




--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-message-dequeuing-hangs-tp4681366p4681579.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message