activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From SledgeHammer <gro...@firstam.com>
Subject Can you help with these exceptions??
Date Tue, 27 Aug 2013 16:55:00 GMT
Having lots of stability issues (5.8.0 on Win64 2008 Server 64GB RAM).
Messages are IBytesMessages, typically around 400k. Typically 3 to 4 queues
running at once. 200 consumers. Checked the log and seeing a TON of
exceptions:

1) INFO   | jvm 1    | 2013/08/26 15:09:32 |  WARN | Transport Connection
to: tcp://x.x.x.x:59339 failed: java.net.SocketException: Software caused
connection abort: socket write error
2) INFO   | jvm 1    | 2013/08/26 15:09:33 |  WARN | Transport Connection
to: tcp://x.x.x.x:59700 failed: java.io.EOFException
3) INFO   | jvm 1    | 2013/08/26 15:09:33 |  WARN | Transport Connection
to: tcp://x.x.x.x:62316 failed: java.net.SocketException: Software caused
connection abort: recv failed
4) INFO   | jvm 1    | 2013/08/26 15:09:33 |  WARN | Transport Connection
to: tcp://x.x.x.x:62316 failed:
org.apache.activemq.transport.InactivityIOException: Cannot send, channel
has already failed: tcp://x.x.x.x:62316

Hundreds / Thousands of these for various IPs and ports.

5) INFO   | jvm 1    | 2013/08/26 16:05:48 |  WARN | Async error occurred:
javax.jms.JMSException: Destination still has an active subscription:
queue://x-caf311ac-edb9-4ff1-8d7b-890d320cece8
INFO   | jvm 1    | 2013/08/26 16:05:48 | javax.jms.JMSException:
Destination still has an active subscription:
queue://x-caf311ac-edb9-4ff1-8d7b-890d320cece8
INFO   | jvm 1    | 2013/08/26 16:05:48 | 	at
org.apache.activemq.broker.region.AbstractRegion.removeDestination(AbstractRegion.java:186)
INFO   | jvm 1    | 2013/08/26 16:05:48 | 	at
org.apache.activemq.broker.jmx.ManagedQueueRegion.removeDestination(ManagedQueueRegion.java:62)
INFO   | jvm 1    | 2013/08/26 16:05:48 | 	at
org.apache.activemq.broker.region.RegionBroker.removeDestination(RegionBroker.java:288)
INFO   | jvm 1    | 2013/08/26 16:05:48 | 	at
org.apache.activemq.broker.region.RegionBroker.removeDestinationInfo(RegionBroker.java:301)
INFO   | jvm 1    | 2013/08/26 16:05:48 | 	at
org.apache.activemq.broker.BrokerFilter.removeDestinationInfo(BrokerFilter.java:221)
INFO   | jvm 1    | 2013/08/26 16:05:48 | 	at
org.apache.activemq.advisory.AdvisoryBroker.removeDestinationInfo(AdvisoryBroker.java:223)
INFO   | jvm 1    | 2013/08/26 16:05:48 | 	at
org.apache.activemq.broker.BrokerFilter.removeDestinationInfo(BrokerFilter.java:221)
INFO   | jvm 1    | 2013/08/26 16:05:48 | 	at
org.apache.activemq.broker.BrokerFilter.removeDestinationInfo(BrokerFilter.java:221)
INFO   | jvm 1    | 2013/08/26 16:05:48 | 	at
org.apache.activemq.broker.MutableBrokerFilter.removeDestinationInfo(MutableBrokerFilter.java:228)
INFO   | jvm 1    | 2013/08/26 16:05:48 | 	at
org.apache.activemq.broker.TransportConnection.processRemoveDestination(TransportConnection.java:537)
INFO   | jvm 1    | 2013/08/26 16:05:48 | 	at
org.apache.activemq.command.DestinationInfo.visit(DestinationInfo.java:124)
INFO   | jvm 1    | 2013/08/26 16:05:48 | 	at
org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:329)
INFO   | jvm 1    | 2013/08/26 16:05:48 | 	at
org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:184)
INFO   | jvm 1    | 2013/08/26 16:05:48 | 	at
org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
INFO   | jvm 1    | 2013/08/26 16:05:48 | 	at
org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
INFO   | jvm 1    | 2013/08/26 16:05:48 | 	at
org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:288)
INFO   | jvm 1    | 2013/08/26 16:05:48 | 	at
org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
INFO   | jvm 1    | 2013/08/26 16:05:48 | 	at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:214)
INFO   | jvm 1    | 2013/08/26 16:05:48 | 	at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
INFO   | jvm 1    | 2013/08/26 16:05:48 | 	at java.lang.Thread.run(Unknown
Source)

6) 
INFO   | jvm 2    | 2013/08/27 09:17:17 |  WARN | Async error occurred:
java.lang.IllegalStateException: Timer already cancelled.
INFO   | jvm 2    | 2013/08/27 09:17:17 | java.lang.IllegalStateException:
Timer already cancelled.
INFO   | jvm 2    | 2013/08/27 09:17:17 | 	at java.util.Timer.sched(Unknown
Source)
INFO   | jvm 2    | 2013/08/27 09:17:17 | 	at
java.util.Timer.schedule(Unknown Source)
INFO   | jvm 2    | 2013/08/27 09:17:17 | 	at
org.apache.activemq.thread.Scheduler.schedualPeriodically(Scheduler.java:50)
INFO   | jvm 2    | 2013/08/27 09:17:17 | 	at
org.apache.activemq.broker.region.Queue.start(Queue.java:889)
INFO   | jvm 2    | 2013/08/27 09:17:17 | 	at
org.apache.activemq.broker.region.AbstractRegion.addDestination(AbstractRegion.java:142)
INFO   | jvm 2    | 2013/08/27 09:17:17 | 	at
org.apache.activemq.broker.region.RegionBroker.addDestination(RegionBroker.java:277)
INFO   | jvm 2    | 2013/08/27 09:17:17 | 	at
org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
INFO   | jvm 2    | 2013/08/27 09:17:17 | 	at
org.apache.activemq.advisory.AdvisoryBroker.addDestination(AdvisoryBroker.java:174)
INFO   | jvm 2    | 2013/08/27 09:17:17 | 	at
org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
INFO   | jvm 2    | 2013/08/27 09:17:17 | 	at
org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
INFO   | jvm 2    | 2013/08/27 09:17:17 | 	at
org.apache.activemq.broker.MutableBrokerFilter.addDestination(MutableBrokerFilter.java:151)
INFO   | jvm 2    | 2013/08/27 09:17:17 | 	at
org.apache.activemq.broker.region.RegionBroker.addProducer(RegionBroker.java:324)
INFO   | jvm 2    | 2013/08/27 09:17:17 | 	at
org.apache.activemq.broker.jmx.ManagedRegionBroker.addProducer(ManagedRegionBroker.java:252)
INFO   | jvm 2    | 2013/08/27 09:17:17 | 	at
org.apache.activemq.broker.BrokerFilter.addProducer(BrokerFilter.java:93)
INFO   | jvm 2    | 2013/08/27 09:17:17 | 	at
org.apache.activemq.advisory.AdvisoryBroker.addProducer(AdvisoryBroker.java:162)
INFO   | jvm 2    | 2013/08/27 09:17:17 | 	at
org.apache.activemq.broker.CompositeDestinationBroker.addProducer(CompositeDestinationBroker.java:56)
INFO   | jvm 2    | 2013/08/27 09:17:17 | 	at
org.apache.activemq.broker.BrokerFilter.addProducer(BrokerFilter.java:93)
INFO   | jvm 2    | 2013/08/27 09:17:17 | 	at
org.apache.activemq.broker.MutableBrokerFilter.addProducer(MutableBrokerFilter.java:99)
INFO   | jvm 2    | 2013/08/27 09:17:17 | 	at
org.apache.activemq.broker.TransportConnection.processAddProducer(TransportConnection.java:566)
INFO   | jvm 2    | 2013/08/27 09:17:17 | 	at
org.apache.activemq.command.ProducerInfo.visit(ProducerInfo.java:105)
INFO   | jvm 2    | 2013/08/27 09:17:17 | 	at
org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:329)
INFO   | jvm 2    | 2013/08/27 09:17:17 | 	at
org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:184)
INFO   | jvm 2    | 2013/08/27 09:17:17 | 	at
org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
INFO   | jvm 2    | 2013/08/27 09:17:17 | 	at
org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
INFO   | jvm 2    | 2013/08/27 09:17:17 | 	at
org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:288)
INFO   | jvm 2    | 2013/08/27 09:17:17 | 	at
org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
INFO   | jvm 2    | 2013/08/27 09:17:17 | 	at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:214)
INFO   | jvm 2    | 2013/08/27 09:17:17 | 	at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
INFO   | jvm 2    | 2013/08/27 09:17:17 | 	at java.lang.Thread.run(Unknown
Source)

7) INFO   | jvm 2    | 2013/08/27 09:17:17 |  WARN | Async error occurred:
java.lang.IllegalStateException: Cannot remove a consumer that had not been
registered: ID:x-w01-49130-635131918368230097-1:0:1:1
INFO   | jvm 2    | 2013/08/27 09:17:17 | java.lang.IllegalStateException:
Cannot remove a consumer that had not been registered:
ID:x-w01-49130-635131918368230097-1:0:1:1
INFO   | jvm 2    | 2013/08/27 09:17:17 | 	at
org.apache.activemq.broker.TransportConnection.processRemoveConsumer(TransportConnection.java:647)
INFO   | jvm 2    | 2013/08/27 09:17:17 | 	at
org.apache.activemq.command.RemoveInfo.visit(RemoveInfo.java:76)
INFO   | jvm 2    | 2013/08/27 09:17:17 | 	at
org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:329)
INFO   | jvm 2    | 2013/08/27 09:17:17 | 	at
org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:184)
INFO   | jvm 2    | 2013/08/27 09:17:17 | 	at
org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
INFO   | jvm 2    | 2013/08/27 09:17:17 | 	at
org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
INFO   | jvm 2    | 2013/08/27 09:17:17 | 	at
org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:288)
INFO   | jvm 2    | 2013/08/27 09:17:17 | 	at
org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
INFO   | jvm 2    | 2013/08/27 09:17:17 | 	at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:214)
INFO   | jvm 2    | 2013/08/27 09:17:17 | 	at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
INFO   | jvm 2    | 2013/08/27 09:17:17 | 	at java.lang.Thread.run(Unknown
Source)

8) INFO   | jvm 2    | 2013/08/27 09:17:17 |  WARN | Transport Connection
to: tcp://x.x.x.x:49129 failed: java.io.EOFException
INFO   | jvm 2    | 2013/08/27 09:18:03 |  WARN | Transport Connection to:
tcp://x.x.x.x:57520 failed: java.net.SocketException: Software caused
connection abort: socket write error
INFO   | jvm 2    | 2013/08/27 09:35:03 |  WARN | /admin/topics.jsp
INFO   | jvm 2    | 2013/08/27 09:35:03 | javax.el.ELException:
java.lang.reflect.UndeclaredThrowableException
INFO   | jvm 2    | 2013/08/27 09:35:03 | 	at
javax.el.BeanELResolver.getValue(BeanELResolver.java:298)
INFO   | jvm 2    | 2013/08/27 09:35:03 | 	at
javax.el.CompositeELResolver.getValue(CompositeELResolver.java:175)
INFO   | jvm 2    | 2013/08/27 09:35:03 | 	at
com.sun.el.parser.AstValue.getValue(AstValue.java:138)
INFO   | jvm 2    | 2013/08/27 09:35:03 | 	at
com.sun.el.ValueExpressionImpl.getValue(ValueExpressionImpl.java:206)
INFO   | jvm 2    | 2013/08/27 09:35:03 | 	at
org.apache.jasper.runtime.PageContextImpl.evaluateExpression(PageContextImpl.java:1001)
INFO   | jvm 2    | 2013/08/27 09:35:03 | 	at
org.apache.jsp.topics_jsp._jspx_meth_c_param_0(org.apache.jsp.topics_jsp:265)
INFO   | jvm 2    | 2013/08/27 09:35:03 | 	at
org.apache.jsp.topics_jsp._jspx_meth_c_url_0(org.apache.jsp.topics_jsp:234)
INFO   | jvm 2    | 2013/08/27 09:35:03 | 	at
org.apache.jsp.topics_jsp._jspx_meth_c_forEach_0(org.apache.jsp.topics_jsp:168)
INFO   | jvm 2    | 2013/08/27 09:35:03 | 	at
org.apache.jsp.topics_jsp._jspService(org.apache.jsp.topics_jsp:110)
INFO   | jvm 2    | 2013/08/27 09:35:03 | 	at
org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:109)
INFO   | jvm 2    | 2013/08/27 09:35:03 | 	at
javax.servlet.http.HttpServlet.service(HttpServlet.java:806)
INFO   | jvm 2    | 2013/08/27 09:35:03 | 	at
org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:389)
INFO   | jvm 2    | 2013/08/27 09:35:03 | 	at
org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:486)
INFO   | jvm 2    | 2013/08/27 09:35:03 | 	at
org.apache.jasper.servlet.JspServlet.service(JspServlet.java:380)
INFO   | jvm 2    | 2013/08/27 09:35:03 | 	at
javax.servlet.http.HttpServlet.service(HttpServlet.java:806)
INFO   | jvm 2    | 2013/08/27 09:35:03 | 	at
org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:652)
INFO   | jvm 2    | 2013/08/27 09:35:03 | 	at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1329)
INFO   | jvm 2    | 2013/08/27 09:35:03 | 	at
org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:83)
INFO   | jvm 2    | 2013/08/27 09:35:03 | 	at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)
INFO   | jvm 2    | 2013/08/27 09:35:03 | 	at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1300)
INFO   | jvm 2    | 2013/08/27 09:35:03 | 	at
org.apache.activemq.web.SessionFilter.doFilter(SessionFilter.java:45)
INFO   | jvm 2    | 2013/08/27 09:35:03 | 	at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1300)
INFO   | jvm 2    | 2013/08/27 09:35:03 | 	at
org.apache.activemq.web.filter.ApplicationContextFilter.doFilter(ApplicationContextFilter.java:102)
INFO   | jvm 2    | 2013/08/27 09:35:03 | 	at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1300)
INFO   | jvm 2    | 2013/08/27 09:35:03 | 	at
com.opensymphony.sitemesh.webapp.SiteMeshFilter.obtainContent(SiteMeshFilter.java:129)
INFO   | jvm 2    | 2013/08/27 09:35:03 | 	at
com.opensymphony.sitemesh.webapp.SiteMeshFilter.doFilter(SiteMeshFilter.java:77)
INFO   | jvm 2    | 2013/08/27 09:35:03 | 	at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1300)
INFO   | jvm 2    | 2013/08/27 09:35:03 | 	at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:445)
INFO   | jvm 2    | 2013/08/27 09:35:03 | 	at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
INFO   | jvm 2    | 2013/08/27 09:35:03 | 	at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:521)
INFO   | jvm 2    | 2013/08/27 09:35:03 | 	at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:227)
INFO   | jvm 2    | 2013/08/27 09:35:03 | 	at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1038)
INFO   | jvm 2    | 2013/08/27 09:35:03 | 	at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:374)
INFO   | jvm 2    | 2013/08/27 09:35:03 | 	at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:189)
INFO   | jvm 2    | 2013/08/27 09:35:03 | 	at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:972)
INFO   | jvm 2    | 2013/08/27 09:35:03 | 	at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
INFO   | jvm 2    | 2013/08/27 09:35:03 | 	at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
INFO   | jvm 2    | 2013/08/27 09:35:03 | 	at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:521)
INFO   | jvm 2    | 2013/08/27 09:35:03 | 	at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
INFO   | jvm 2    | 2013/08/27 09:35:03 | 	at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
INFO   | jvm 2    | 2013/08/27 09:35:03 | 	at
org.eclipse.jetty.server.Server.handle(Server.java:363)
INFO   | jvm 2    | 2013/08/27 09:35:03 | 	at
org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:483)
INFO   | jvm 2    | 2013/08/27 09:35:03 | 	at
org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:920)
INFO   | jvm 2    | 2013/08/27 09:35:03 | 	at
org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:982)
INFO   | jvm 2    | 2013/08/27 09:35:03 | 	at
org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:635)
INFO   | jvm 2    | 2013/08/27 09:35:03 | 	at
org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
INFO   | jvm 2    | 2013/08/27 09:35:03 | 	at
org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
INFO   | jvm 2    | 2013/08/27 09:35:03 | 	at
org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:627)
INFO   | jvm 2    | 2013/08/27 09:35:03 | 	at
org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:51)
INFO   | jvm 2    | 2013/08/27 09:35:03 | 	at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
INFO   | jvm 2    | 2013/08/27 09:35:03 | 	at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
INFO   | jvm 2    | 2013/08/27 09:35:03 | 	at java.lang.Thread.run(Unknown
Source)
INFO   | jvm 2    | 2013/08/27 09:35:03 | Caused by:
java.lang.reflect.UndeclaredThrowableException
INFO   | jvm 2    | 2013/08/27 09:35:03 | 	at
com.sun.proxy.$Proxy6.getName(Unknown Source)
INFO   | jvm 2    | 2013/08/27 09:35:03 | 	at
sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
INFO   | jvm 2    | 2013/08/27 09:35:03 | 	at
sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
INFO   | jvm 2    | 2013/08/27 09:35:03 | 	at
java.lang.reflect.Method.invoke(Unknown Source)
INFO   | jvm 2    | 2013/08/27 09:35:03 | 	at
javax.el.BeanELResolver.getValue(BeanELResolver.java:293)
INFO   | jvm 2    | 2013/08/27 09:35:03 | 	... 51 more
INFO   | jvm 2    | 2013/08/27 09:35:03 | Caused by:
javax.management.InstanceNotFoundException:
org.apache.activemq:type=Broker,brokerName=dtresna9shyp029.datatree.local,destinationType=Topic,destinationName=ActiveMQ.Advisory.Consumer.Queue.x-3b0e195e-b400-420c-8d30-1c641b6dffa8
INFO   | jvm 2    | 2013/08/27 09:35:03 | 	at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(Unknown
Source)
INFO   | jvm 2    | 2013/08/27 09:35:03 | 	at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(Unknown
Source)
INFO   | jvm 2    | 2013/08/27 09:35:03 | 	at
com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(Unknown Source)
INFO   | jvm 2    | 2013/08/27 09:35:03 | 	at
javax.management.MBeanServerInvocationHandler.invoke(Unknown Source)
INFO   | jvm 2    | 2013/08/27 09:35:03 | 	... 56 more

9) also gettign some about:

INFO   | jvm 2    | 2013/08/27 09:39:29 |
java.lang.UnsupportedOperationException: Possible CSRF attack

What in the world is going on here?




--
View this message in context: http://activemq.2283324.n4.nabble.com/Can-you-help-with-these-exceptions-tp4670752.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Mime
View raw message