activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gary Tully <gary.tu...@gmail.com>
Subject Re: Errors under high-load (ActiveMQ 5.3 + JBoss 5.1)
Date Mon, 21 Dec 2009 16:38:21 GMT
please file issues but the key component will be some sort of test case that
makes it easy to reproduce.

2009/12/21 Leonardo Lima <leomrlima@gmail.com>

> Hello.
>
> As I continue to try and debug my problem, I see that both errors are
> because of ActiveMQ.
>
> There are three kinds of stacktraces from ActiveMQ Logs:
>
> 2009-12-21 12:19:38,296 | WARN  | POST COMMIT FAILED:  |
> org.apache.activemq.transaction.XATransaction | NIO Transport Thread
> java.util.ConcurrentModificationException
>    at
> java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
>    at java.util.AbstractList$Itr.next(AbstractList.java:343)
>    at
>
> org.apache.activemq.transaction.Transaction.fireAfterCommit(Transaction.java:86)
>    at
>
> org.apache.activemq.transaction.XATransaction.doPostCommit(XATransaction.java:113)
>    at
> org.apache.activemq.transaction.XATransaction.commit(XATransaction.java:68)
>    at
>
> org.apache.activemq.broker.TransactionBroker.commitTransaction(TransactionBroker.java:170)
>    at
>
> org.apache.activemq.broker.BrokerFilter.commitTransaction(BrokerFilter.java:94)
>    at
>
> org.apache.activemq.broker.MutableBrokerFilter.commitTransaction(MutableBrokerFilter.java:101)
>    at
>
> org.apache.activemq.broker.TransportConnection.processCommitTransactionOnePhase(TransportConnection.java:406)
>    at
> org.apache.activemq.command.TransactionInfo.visit(TransactionInfo.java:100)
>    at
>
> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:297)
>    at
>
> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:175)
>    at
>
> org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
>    at
>
> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
>    at
>
> org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:210)
>    at
>
> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:84)
>    at
>
> org.apache.activemq.transport.nio.NIOTransport.serviceRead(NIOTransport.java:128)
>    at
>
> org.apache.activemq.transport.nio.NIOTransport.access$000(NIOTransport.java:44)
>    at
>
> org.apache.activemq.transport.nio.NIOTransport$1.onSelect(NIOTransport.java:68)
>    at
>
> org.apache.activemq.transport.nio.SelectorSelection.onSelect(SelectorSelection.java:78)
>    at
>
> org.apache.activemq.transport.nio.SelectorWorker$1.run(SelectorWorker.java:103)
>    at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>    at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>    at java.lang.Thread.run(Thread.java:619)
>
> 2009-12-21 12:17:10,281 | WARN  | Async error occurred:
> javax.transaction.xa.XAException: Transaction
>
> 'XID:131075:312d2d33663537633766663a663839353a34623266326262613a313663653763:2d33663537633766663a663839353a34623266326262613a313663653764'
> has not been started. |
> org.apache.activemq.broker.TransportConnection.Service | NIO Transport
> Thread
> javax.transaction.xa.XAException: Transaction
>
> 'XID:131075:312d2d33663537633766663a663839353a34623266326262613a313663653763:2d33663537633766663a663839353a34623266326262613a313663653764'
> has not been started.
>    at
>
> org.apache.activemq.broker.TransactionBroker.getTransaction(TransactionBroker.java:266)
>    at
>
> org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:208)
>    at
>
> org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:95)
>    at
>
> org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:133)
>    at
>
> org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:443)
>    at
> org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:631)
>     at
>
> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:297)
>    at
>
> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:175)
>    at
>
> org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
>    at
>
> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
>    at
>
> org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:210)
>    at
>
> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:84)
>    at
>
> org.apache.activemq.transport.nio.NIOTransport.serviceRead(NIOTransport.java:128)
>    at
>
> org.apache.activemq.transport.nio.NIOTransport.access$000(NIOTransport.java:44)
>    at
>
> org.apache.activemq.transport.nio.NIOTransport$1.onSelect(NIOTransport.java:68)
>    at
>
> org.apache.activemq.transport.nio.SelectorSelection.onSelect(SelectorSelection.java:78)
>    at
>
> org.apache.activemq.transport.nio.SelectorWorker$1.run(SelectorWorker.java:103)
>    at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>    at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>    at java.lang.Thread.run(Thread.java:619)
>
> 2009-12-20 19:45:50,406 | WARN  | Async error occurred:
> java.lang.ArrayIndexOutOfBoundsException: -1 |
> org.apache.activemq.broker.TransportConnection.Service | NIO Transport
> Thread
> java.lang.ArrayIndexOutOfBoundsException: -1
>    at java.util.ArrayList.get(ArrayList.java:324)
>    at
>
> org.apache.activemq.store.kahadb.KahaDBStore.createTransactionInfo(KahaDBStore.java:552)
>    at
>
> org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.removeMessage(KahaDBStore.java:152)
>    at org.apache.activemq.broker.region.Queue.acknowledge(Queue.java:609)
>    at
> org.apache.activemq.broker.region.Queue.removeMessage(Queue.java:1237)
>    at
>
> org.apache.activemq.broker.region.QueueSubscription.acknowledge(QueueSubscription.java:60)
>    at
>
> org.apache.activemq.broker.region.PrefetchSubscription.acknowledge(PrefetchSubscription.java:250)
>    at
>
> org.apache.activemq.broker.region.AbstractRegion.acknowledge(AbstractRegion.java:369)
>    at
>
> org.apache.activemq.broker.region.RegionBroker.acknowledge(RegionBroker.java:470)
>    at
>
> org.apache.activemq.broker.TransactionBroker.acknowledge(TransactionBroker.java:194)
>    at
> org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:74)
>    at
>
> org.apache.activemq.broker.MutableBrokerFilter.acknowledge(MutableBrokerFilter.java:85)
>    at
>
> org.apache.activemq.broker.TransportConnection.processMessageAck(TransportConnection.java:449)
>    at org.apache.activemq.command.MessageAck.visit(MessageAck.java:205)
>     at
>
> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:297)
>    at
>
> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:175)
>    at
>
> org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
>    at
>
> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
>    at
>
> org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:210)
>    at
>
> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:84)
>    at
>
> org.apache.activemq.transport.nio.NIOTransport.serviceRead(NIOTransport.java:128)
>    at
>
> org.apache.activemq.transport.nio.NIOTransport.access$000(NIOTransport.java:44)
>    at
>
> org.apache.activemq.transport.nio.NIOTransport$1.onSelect(NIOTransport.java:68)
>    at
>
> org.apache.activemq.transport.nio.SelectorSelection.onSelect(SelectorSelection.java:78)
>    at
>
> org.apache.activemq.transport.nio.SelectorWorker$1.run(SelectorWorker.java:103)
>    at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>    at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>    at java.lang.Thread.run(Thread.java:619)
>
> I'm thinking about filing bugs for each of one, should I?
>
> Thank you very much.
> Leonardo.
>
> On Mon, Dec 21, 2009 at 7:35 AM, Leonardo Lima <leomrlima@gmail.com>
> wrote:
>
> > Hello.
> >
> > I'm stress-testing this scenario: about 15000 messages are sent in about
> 10
> > min (~25 msg/sec) to a standalone broker (using nio transport). Messages
> are
> > consumed via MDB in a Jboss 5.1, configured with the activemq-rar
> > instructions found on the website (
> > http://activemq.apache.org/integrating-apache-activemq-with-jboss.html).
> > The MDBs are also inserting data in a database, and everything is under
> XA
> > (or so it seems from the errors...). I'm running everything in a Windows
> > 2008 R2 and Sun's 64-bit JVM.
> >
> > Everything seems to run fine, but after the messages are sent, some (~50
> > messages) about are "lingering" in the queue (Number Of Pending Messages
> is
> > bigger than 0). But no one consumer is getting messages anymore and I
> can't
> > browse them.
> >
> > Checking for errors, I see two issues. Sometimes, I get a
> > "javax.transaction.xa.XAException: Transaction has not been started.".
> That
> > shows at both JBoss and ActiveMQ. I've searched about it and found little
> > info on how to debug it. I think it must be a configuration mismatch
> about
> > quantities of avaiable MDBs/Connections/Transaction/Pool or something
> like
> > that, but I don't know. And I believe ActiveMQ would retry to dispatch
> this
> > message again, so this wouldn't be a big deal.
> >
> > And sometimes I get a "POST COMMIT FAILED"
> > java.util.ConcurrentModificationException at ActiveMQ. I couldn't find
> this
> > error anywhere else, and I don't know what else can I do about it. Is it
> a
> > bug? Maybe it has something to do with this pending messages. Here goes
> the
> > stacktrace:
> >
> > 2009-12-21 07:05:31,522 | WARN  | POST COMMIT FAILED:  |
> > org.apache.activemq.transaction.XATransaction | NIO Transport Thread
> > java.util.ConcurrentModificationException
> >     at
> > java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
> >     at java.util.AbstractList$Itr.next(AbstractList.java:343)
> >     at
> >
> org.apache.activemq.transaction.Transaction.fireAfterCommit(Transaction.java:86)
> >     at
> >
> org.apache.activemq.transaction.XATransaction.doPostCommit(XATransaction.java:113)
> >     at
> >
> org.apache.activemq.transaction.XATransaction.commit(XATransaction.java:68)
> >     at
> >
> org.apache.activemq.broker.TransactionBroker.commitTransaction(TransactionBroker.java:170)
> >     at
> >
> org.apache.activemq.broker.BrokerFilter.commitTransaction(BrokerFilter.java:94)
> >     at
> >
> org.apache.activemq.broker.MutableBrokerFilter.commitTransaction(MutableBrokerFilter.java:101)
> >     at
> >
> org.apache.activemq.broker.TransportConnection.processCommitTransactionOnePhase(TransportConnection.java:406)
> >     at
> >
> org.apache.activemq.command.TransactionInfo.visit(TransactionInfo.java:100)
> >     at
> >
> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:297)
> >     at
> >
> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:175)
> >     at
> >
> org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
> >     at
> >
> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
> >     at
> >
> org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:210)
> >     at
> >
> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:84)
> >     at
> >
> org.apache.activemq.transport.nio.NIOTransport.serviceRead(NIOTransport.java:128)
> >     at
> >
> org.apache.activemq.transport.nio.NIOTransport.access$000(NIOTransport.java:44)
> >     at
> >
> org.apache.activemq.transport.nio.NIOTransport$1.onSelect(NIOTransport.java:68)
> >     at
> >
> org.apache.activemq.transport.nio.SelectorSelection.onSelect(SelectorSelection.java:78)
> >     at
> >
> org.apache.activemq.transport.nio.SelectorWorker$1.run(SelectorWorker.java:103)
> >     at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> >     at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> >     at java.lang.Thread.run(Thread.java:619)
> >
> > My main concern is that I don't know if there really is something pending
> > and how should I avoid this.
> >
> > Could you please take a look at this and give me pointers on how to
> > proceed?
> >
> > Thank you very much.
> > Leo
> >
>



-- 
http://blog.garytully.com

Open Source Integration
http://fusesource.com

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message