activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Leonardo Lima <leomrl...@gmail.com>
Subject Re: Errors under high-load (ActiveMQ 5.3 + JBoss 5.1)
Date Mon, 21 Dec 2009 14:11:50 GMT
Hello.

As I continue to try and debug my problem, I see that both errors are
because of ActiveMQ.

There are three kinds of stacktraces from ActiveMQ Logs:

2009-12-21 12:19:38,296 | WARN  | POST COMMIT FAILED:  |
org.apache.activemq.transaction.XATransaction | NIO Transport Thread
java.util.ConcurrentModificationException
    at
java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
    at java.util.AbstractList$Itr.next(AbstractList.java:343)
    at
org.apache.activemq.transaction.Transaction.fireAfterCommit(Transaction.java:86)
    at
org.apache.activemq.transaction.XATransaction.doPostCommit(XATransaction.java:113)
    at
org.apache.activemq.transaction.XATransaction.commit(XATransaction.java:68)
    at
org.apache.activemq.broker.TransactionBroker.commitTransaction(TransactionBroker.java:170)
    at
org.apache.activemq.broker.BrokerFilter.commitTransaction(BrokerFilter.java:94)
    at
org.apache.activemq.broker.MutableBrokerFilter.commitTransaction(MutableBrokerFilter.java:101)
    at
org.apache.activemq.broker.TransportConnection.processCommitTransactionOnePhase(TransportConnection.java:406)
    at
org.apache.activemq.command.TransactionInfo.visit(TransactionInfo.java:100)
    at
org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:297)
    at
org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:175)
    at
org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
    at
org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
    at
org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:210)
    at
org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:84)
    at
org.apache.activemq.transport.nio.NIOTransport.serviceRead(NIOTransport.java:128)
    at
org.apache.activemq.transport.nio.NIOTransport.access$000(NIOTransport.java:44)
    at
org.apache.activemq.transport.nio.NIOTransport$1.onSelect(NIOTransport.java:68)
    at
org.apache.activemq.transport.nio.SelectorSelection.onSelect(SelectorSelection.java:78)
    at
org.apache.activemq.transport.nio.SelectorWorker$1.run(SelectorWorker.java:103)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:619)

2009-12-21 12:17:10,281 | WARN  | Async error occurred:
javax.transaction.xa.XAException: Transaction
'XID:131075:312d2d33663537633766663a663839353a34623266326262613a313663653763:2d33663537633766663a663839353a34623266326262613a313663653764'
has not been started. |
org.apache.activemq.broker.TransportConnection.Service | NIO Transport
Thread
javax.transaction.xa.XAException: Transaction
'XID:131075:312d2d33663537633766663a663839353a34623266326262613a313663653763:2d33663537633766663a663839353a34623266326262613a313663653764'
has not been started.
    at
org.apache.activemq.broker.TransactionBroker.getTransaction(TransactionBroker.java:266)
    at
org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:208)
    at
org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:95)
    at
org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:133)
    at
org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:443)
    at
org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:631)
    at
org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:297)
    at
org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:175)
    at
org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
    at
org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
    at
org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:210)
    at
org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:84)
    at
org.apache.activemq.transport.nio.NIOTransport.serviceRead(NIOTransport.java:128)
    at
org.apache.activemq.transport.nio.NIOTransport.access$000(NIOTransport.java:44)
    at
org.apache.activemq.transport.nio.NIOTransport$1.onSelect(NIOTransport.java:68)
    at
org.apache.activemq.transport.nio.SelectorSelection.onSelect(SelectorSelection.java:78)
    at
org.apache.activemq.transport.nio.SelectorWorker$1.run(SelectorWorker.java:103)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:619)

2009-12-20 19:45:50,406 | WARN  | Async error occurred:
java.lang.ArrayIndexOutOfBoundsException: -1 |
org.apache.activemq.broker.TransportConnection.Service | NIO Transport
Thread
java.lang.ArrayIndexOutOfBoundsException: -1
    at java.util.ArrayList.get(ArrayList.java:324)
    at
org.apache.activemq.store.kahadb.KahaDBStore.createTransactionInfo(KahaDBStore.java:552)
    at
org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.removeMessage(KahaDBStore.java:152)
    at org.apache.activemq.broker.region.Queue.acknowledge(Queue.java:609)
    at
org.apache.activemq.broker.region.Queue.removeMessage(Queue.java:1237)
    at
org.apache.activemq.broker.region.QueueSubscription.acknowledge(QueueSubscription.java:60)
    at
org.apache.activemq.broker.region.PrefetchSubscription.acknowledge(PrefetchSubscription.java:250)
    at
org.apache.activemq.broker.region.AbstractRegion.acknowledge(AbstractRegion.java:369)
    at
org.apache.activemq.broker.region.RegionBroker.acknowledge(RegionBroker.java:470)
    at
org.apache.activemq.broker.TransactionBroker.acknowledge(TransactionBroker.java:194)
    at
org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:74)
    at
org.apache.activemq.broker.MutableBrokerFilter.acknowledge(MutableBrokerFilter.java:85)
    at
org.apache.activemq.broker.TransportConnection.processMessageAck(TransportConnection.java:449)
    at org.apache.activemq.command.MessageAck.visit(MessageAck.java:205)
    at
org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:297)
    at
org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:175)
    at
org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
    at
org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
    at
org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:210)
    at
org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:84)
    at
org.apache.activemq.transport.nio.NIOTransport.serviceRead(NIOTransport.java:128)
    at
org.apache.activemq.transport.nio.NIOTransport.access$000(NIOTransport.java:44)
    at
org.apache.activemq.transport.nio.NIOTransport$1.onSelect(NIOTransport.java:68)
    at
org.apache.activemq.transport.nio.SelectorSelection.onSelect(SelectorSelection.java:78)
    at
org.apache.activemq.transport.nio.SelectorWorker$1.run(SelectorWorker.java:103)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:619)

I'm thinking about filing bugs for each of one, should I?

Thank you very much.
Leonardo.

On Mon, Dec 21, 2009 at 7:35 AM, Leonardo Lima <leomrlima@gmail.com> wrote:

> Hello.
>
> I'm stress-testing this scenario: about 15000 messages are sent in about 10
> min (~25 msg/sec) to a standalone broker (using nio transport). Messages are
> consumed via MDB in a Jboss 5.1, configured with the activemq-rar
> instructions found on the website (
> http://activemq.apache.org/integrating-apache-activemq-with-jboss.html).
> The MDBs are also inserting data in a database, and everything is under XA
> (or so it seems from the errors...). I'm running everything in a Windows
> 2008 R2 and Sun's 64-bit JVM.
>
> Everything seems to run fine, but after the messages are sent, some (~50
> messages) about are "lingering" in the queue (Number Of Pending Messages is
> bigger than 0). But no one consumer is getting messages anymore and I can't
> browse them.
>
> Checking for errors, I see two issues. Sometimes, I get a
> "javax.transaction.xa.XAException: Transaction has not been started.". That
> shows at both JBoss and ActiveMQ. I've searched about it and found little
> info on how to debug it. I think it must be a configuration mismatch about
> quantities of avaiable MDBs/Connections/Transaction/Pool or something like
> that, but I don't know. And I believe ActiveMQ would retry to dispatch this
> message again, so this wouldn't be a big deal.
>
> And sometimes I get a "POST COMMIT FAILED"
> java.util.ConcurrentModificationException at ActiveMQ. I couldn't find this
> error anywhere else, and I don't know what else can I do about it. Is it a
> bug? Maybe it has something to do with this pending messages. Here goes the
> stacktrace:
>
> 2009-12-21 07:05:31,522 | WARN  | POST COMMIT FAILED:  |
> org.apache.activemq.transaction.XATransaction | NIO Transport Thread
> java.util.ConcurrentModificationException
>     at
> java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
>     at java.util.AbstractList$Itr.next(AbstractList.java:343)
>     at
> org.apache.activemq.transaction.Transaction.fireAfterCommit(Transaction.java:86)
>     at
> org.apache.activemq.transaction.XATransaction.doPostCommit(XATransaction.java:113)
>     at
> org.apache.activemq.transaction.XATransaction.commit(XATransaction.java:68)
>     at
> org.apache.activemq.broker.TransactionBroker.commitTransaction(TransactionBroker.java:170)
>     at
> org.apache.activemq.broker.BrokerFilter.commitTransaction(BrokerFilter.java:94)
>     at
> org.apache.activemq.broker.MutableBrokerFilter.commitTransaction(MutableBrokerFilter.java:101)
>     at
> org.apache.activemq.broker.TransportConnection.processCommitTransactionOnePhase(TransportConnection.java:406)
>     at
> org.apache.activemq.command.TransactionInfo.visit(TransactionInfo.java:100)
>     at
> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:297)
>     at
> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:175)
>     at
> org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
>     at
> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
>     at
> org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:210)
>     at
> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:84)
>     at
> org.apache.activemq.transport.nio.NIOTransport.serviceRead(NIOTransport.java:128)
>     at
> org.apache.activemq.transport.nio.NIOTransport.access$000(NIOTransport.java:44)
>     at
> org.apache.activemq.transport.nio.NIOTransport$1.onSelect(NIOTransport.java:68)
>     at
> org.apache.activemq.transport.nio.SelectorSelection.onSelect(SelectorSelection.java:78)
>     at
> org.apache.activemq.transport.nio.SelectorWorker$1.run(SelectorWorker.java:103)
>     at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>     at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>     at java.lang.Thread.run(Thread.java:619)
>
> My main concern is that I don't know if there really is something pending
> and how should I avoid this.
>
> Could you please take a look at this and give me pointers on how to
> proceed?
>
> Thank you very much.
> Leo
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message