activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rob Davies <rajdav...@gmail.com>
Subject Re: Standalone server restarting itself every 5 minutes - java.lang.IllegalStateException
Date Fri, 31 Jul 2009 07:59:51 GMT
ooh - sounds like you have more than one broker running in your  
environment - something is creating a network connection to your broker

On 30 Jul 2009, at 18:58, GaryG wrote:

>
> Thank you for pointing that out!  Since we only have one broker, I  
> disabled
> the multicast discovery.
>
> Unfortunately, that didn't fix the problem.  The restarts are still
> happening.  Sometimes they occur within few minutes of startup,  
> sometimes
> takes 4-5 minutes.  The only relevant info I see in "activemq.log"  
> is this:
>
> 2009-07-30 17:42:54,652 [MQ ShutdownHook] INFO  BrokerService
> - ActiveMQ Message Broker (ai_jms,  
> ID:server2-53314-1248975478282-0:0) is
> shutting down
> 2009-07-30 17:42:54,653 [MQ ShutdownHook] DEBUG BrokerService
> - Caught exception, must be shutting down:  
> java.lang.IllegalStateException:
> Shutdown in progress
> 2009-07-30 17:42:54,657 [MQ ShutdownHook] INFO  NetworkConnector
> - Network Connector local-nc Stopped
> 2009-07-30 17:42:56,089 [31.140.18:56800]
>
> Which makes me thing that something is sending a kill command to the
> process, but I have no idea what that could be.
>
> _Gary
>
>
> rajdavies wrote:
>>
>> first - try disabling multicast discovery in the activemq  
>> configuration
>>
>> On 29 Jul 2009, at 22:17, GaryG wrote:
>>
>>>
>>> We've recently updated to ActiveMQ 5.2.0, and are running it as a
>>> standalone
>>> server, with a config file very similar to the basic example
>>> provided out of
>>> the box.  There has been no code change in the Java code that talks
>>> to the
>>> AMQ server.  Yet, once started up, the ActiveMQ server process is
>>> restarting
>>> itself every 5 minutes.
>>>
>>> Looking at the "activemq.log", I see the following:
>>> ===============
>>> .
>>> .
>>> .
>>> 2009-07-29 20:48:01,949 [31.140.17:41258] DEBUG AbstractRegion
>>> - Adding destination: topic://
>>> ActiveMQ.Advisory.Producer.Topic.dr01b_wf1
>>> 2009-07-29 20:48:01,951 [31.140.17:41258] DEBUG AbstractRegion
>>> - Adding destination:
>>> topic://ActiveMQ.Advisory.Producer.Topic.WorkerFarmControl
>>> 2009-07-29 20:48:01,952 [31.140.17:41258] DEBUG AbstractRegion
>>> - Adding destination: topic://
>>> ActiveMQ.Advisory.Producer.Topic.dr01a_wf1
>>> 2009-07-29 20:48:01,956 [31.140.17:41258] DEBUG AbstractRegion
>>> - Adding destination: topic://ControllerCCTopicSend
>>> 2009-07-29 20:48:01,957 [31.140.17:41258] DEBUG
>>> JournalPersistenceAdapter
>>> - Waking for checkpoint to complete.
>>> 2009-07-29 20:48:01,957 [eckpoint Worker] DEBUG
>>> JournalPersistenceAdapter
>>> - Checkpoint started.
>>> 2009-07-29 20:48:01,979 [eckpoint Worker] DEBUG
>>> JournalPersistenceAdapter
>>> - Checkpoint done.
>>> 2009-07-29 20:48:01,984 [31.140.17:41258] DEBUG AbstractRegion
>>> - Adding destination:
>>> topic://ActiveMQ.Advisory.Producer.Topic.ControllerCCTopicSend
>>> 2009-07-29 20:52:55,790 [MQ ShutdownHook] INFO  BrokerService
>>> - ActiveMQ Message Broker (ai_jms,
>>> ID:server2-55452-1248900479410-0:0) is
>>> shutting down
>>> 2009-07-29 20:52:55,791 [MQ ShutdownHook] DEBUG BrokerService
>>> - Caught exception, must be shutting down:
>>> java.lang.IllegalStateException:
>>> Shutdown in progress
>>> 2009-07-29 20:52:55,796 [MQ ShutdownHook] INFO  NetworkConnector
>>> - Network Connector local-nc Stopped
>>> 2009-07-29 20:52:57,294 [31.140.18:56300] DEBUG TransportConnection
>>> - Stopping connection: /1.1.1.18:56300
>>> 2009-07-29 20:52:57,294 [31.140.18:56300] DEBUG TcpTransport
>>> - Stopping transport tcp:///1.1.1.18:56300
>>> 2009-07-29 20:52:57,297 [31.140.18:56300] DEBUG TransportConnection
>>> - Stopped transport: /1.1.1.18:56300
>>> 2009-07-29 20:52:57,298 [31.140.18:56300] DEBUG TransportConnection
>>> - Cleaning up connection resources: /1.1.1.18:56300
>>> 2009-07-29 20:52:57,299 [31.140.18:56300] DEBUG
>>> JournalPersistenceAdapter
>>> - Waking for checkpoint to complete.
>>> 2009-07-29 20:52:57,299 [eckpoint Worker] DEBUG
>>> JournalPersistenceAdapter
>>> - Checkpoint started.
>>> 2009-07-29 20:52:57,331 [eckpoint Worker] DEBUG
>>> JournalPersistenceAdapter
>>> - Checkpoint done.
>>> 2009-07-29 20:52:57,338 [31.140.18:56300] DEBUG AbstractRegion
>>> - Removing consumer: ID:server2-49559-1248899931358-0:2:-1:1
>>> .
>>> .
>>> .
>>> -------------
>>>
>>> So you can see for a while, the broker's adding topics and such, and
>>> then
>>> all of a sudden it gets an "IllegalStateException", and just  
>>> restarts.
>>>
>>>
>>> On the clients, when this happens we get this error:
>>>
>>> --------------
>>> 2009-07-29 20:52:48,550 [ActiveMQ Session Task] DEBUG jms_comm -
>>> ActiveMQMessageReceiver[dr01b_wf1] got msg from = STG_AIC1,
>>> jmsMsgID=ID:server1-41646-1248899975358-0:2:6:1:224
>>> 2009-07-29 20:52:48,550 [ActiveMQ Session Task] DEBUG jms_comm -
>>> ActiveMQMessageReceiver[dr01b_wf1] sent processed msg to
>>> messageListener!
>>> 2009-07-29 20:52:49,896 [ActiveMQ Session Task] DEBUG jms_comm -
>>> ActiveMQMessageReceiver[dr01b_wf1] got msg from = STG_AIC1,
>>> jmsMsgID=ID:server1-41646-1248899975358-0:2:6:1:225
>>> 2009-07-29 20:52:49,896 [ActiveMQ Session Task] DEBUG jms_comm -
>>> ActiveMQMessageReceiver[dr01b_wf1] sent processed msg to
>>> messageListener!
>>> 2009-07-29 20:52:57,865 [ActiveMQ Connection Worker:
>>> tcp://server2/1.1.1.18:61616] ERROR jms_comm -  
>>> FactoryUtil.onException
>>> javax.jms.JMSException: java.io.EOFException
>>>       at
>>> org
>>> .apache
>>> .activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java: 
>>> 49)
>>>       at
>>> org
>>> .apache
>>> .activemq
>>> .ActiveMQConnection.onAsyncException(ActiveMQConnection.java:1773)
>>>       at
>>> org
>>> .apache
>>> .activemq.ActiveMQConnection.onException(ActiveMQConnection.java: 
>>> 1790)
>>>       at
>>> org
>>> .apache
>>> .activemq 
>>> .transport.TransportFilter.onException(TransportFilter.java:
>>> 99)
>>>       at
>>> org
>>> .apache
>>> .activemq
>>> .transport.ResponseCorrelator.onException(ResponseCorrelator.java: 
>>> 114)
>>>       at
>>> org
>>> .apache
>>> .activemq 
>>> .transport.TransportFilter.onException(TransportFilter.java:
>>> 99)
>>>       at
>>> org
>>> .apache
>>> .activemq 
>>> .transport.TransportFilter.onException(TransportFilter.java:
>>> 99)
>>>       at
>>> org
>>> .apache
>>> .activemq
>>> .transport
>>> .WireFormatNegotiator.onException(WireFormatNegotiator.java:152)
>>>       at
>>> org
>>> .apache
>>> .activemq
>>> .transport.InactivityMonitor.onException(InactivityMonitor.java:244)
>>>       at
>>> org
>>> .apache
>>> .activemq
>>> .transport.TransportSupport.onException(TransportSupport.java:96)
>>>       at
>>> org 
>>> .apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:
>>> 189)
>>>       at java.lang.Thread.run(Thread.java:619)
>>> Caused by: java.io.EOFException
>>>       at java.io.DataInputStream.readInt(DataInputStream.java:375)
>>>       at
>>> org
>>> .apache
>>> .activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
>>>       at
>>> org
>>> .apache
>>> .activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:
>>> 210)
>>>       at
>>> org
>>> .apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:
>>> 202)
>>>       at
>>> org 
>>> .apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:
>>> 185)
>>>       ... 1 more
>>> 2009-07-29 20:52:57,895 [ActiveMQ Connection Worker:
>>> tcp://server2.mss.iss.net/1.1.1.18:61616] DEBUG jms_comm - JMS
>>> Exception
>>> occured. Server is down.
>>> 2009-07-29 20:52:58,263 [Thread-27] ERROR jms_comm.log -
>>> ActiveMQMessageSender[ControllerTaskStatusTopic] Error sending
>>> message:
>>> javax.jms.JMSException: Channel was inactive for too long:
>>> server2/1.1.1.18:61616
>>> --------------
>>>
>>>
>>> Our "activemq.xml" file is pretty simple - it's almost exactly the
>>> default
>>> config with just some minor modifications:
>>> ----------------------
>>>
>>> <!--
>>>   Licensed to the Apache Software Foundation (ASF) under one or more
>>>   contributor license agreements.  See the NOTICE file distributed
>>> with
>>>   this work for additional information regarding copyright  
>>> ownership.
>>>   The ASF licenses this file to You under the Apache License,
>>> Version 2.0
>>>   (the "License"); you may not use this file except in compliance
>>> with
>>>   the License.  You may obtain a copy of the License at
>>>
>>>   http://www.apache.org/licenses/LICENSE-2.0
>>>
>>>   Unless required by applicable law or agreed to in writing,  
>>> software
>>>   distributed under the License is distributed on an "AS IS" BASIS,
>>>   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
>>> implied.
>>>   See the License for the specific language governing permissions  
>>> and
>>>   limitations under the License.
>>> -->
>>> <!-- START SNIPPET: example -->
>>> <beans
>>> xmlns="http://www.springframework.org/schema/beans"
>>> xmlns:amq="http://activemq.apache.org/schema/core"
>>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>>> xsi:schemaLocation="http://www.springframework.org/schema/beans
>>> http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
>>> http://activemq.apache.org/schema/core
>>> http://activemq.apache.org/schema/core/activemq-core.xsd
>>> http://activemq.apache.org/camel/schema/spring
>>> http://activemq.apache.org/camel/schema/spring/camel-spring.xsd">
>>>
>>> <!-- Allows us to use system properties as variables in this
>>> configuration
>>> file -->
>>> <bean
>>> class
>>> =
>>> "org
>>> .springframework 
>>> .beans.factory.config.PropertyPlaceholderConfigurer">
>>>   <property name="locations">
>>>     <value>file:///${activemq.base}/conf/credentials.properties</
>>> value>
>>>   </property>
>>> </bean>
>>>
>>> <broker xmlns="http://activemq.apache.org/schema/core"
>>> brokerName="ai_jms"
>>> useJmx="true" dataDirectory="${activemq.base}/data">
>>>
>>>   <!-- The store and forward broker networks ActiveMQ will listen
>>> to -->
>>>   <networkConnectors>
>>>     <!-- by default just auto discover the other brokers -->
>>>     <!--<networkConnector name="default-nc" uri="multicast://
>>> default"/>
>>> -->
>>>     <!-- Example of a static configuration: -->
>>>     <networkConnector name="local-nc"
>>> uri="static://(tcp://localhost:61616)"/>
>>>   </networkConnectors>
>>>
>>>   <persistenceAdapter>
>>>     <journaledJDBC journalLogFiles="10"
>>> dataDirectory="${activemq.base}/activemq-data" dataSource="#derby-
>>> ds"/>
>>>   </persistenceAdapter>
>>>
>>>
>>>   <!--  The maximum about of space the broker will use before
>>> slowing down
>>> producers -->
>>>   <systemUsage>
>>>     <systemUsage>
>>>       <memoryUsage>
>>>         <memoryUsage limit="512 mb"/>
>>>       </memoryUsage>
>>>       <storeUsage>
>>>         <storeUsage limit="1 gb" name="foo"/>
>>>       </storeUsage>
>>>       <tempUsage>
>>>         <tempUsage limit="100 mb"/>
>>>       </tempUsage>
>>>     </systemUsage>
>>>   </systemUsage>
>>>
>>>   <!-- The transport connectors ActiveMQ will listen to -->
>>>   <transportConnectors>
>>>     <transportConnector name="openwire" uri="tcp://localhost:61616"
>>> discoveryUri="multicast://default"/>
>>>   </transportConnectors>
>>>
>>> </broker>
>>>
>>> <!-- An embedded servlet engine for serving up the Admin console -->
>>> <jetty xmlns="http://mortbay.com/schemas/jetty/1.0">
>>>   <connectors>
>>>     <nioConnector port="8161"/>
>>>   </connectors>
>>>
>>>   <handlers>
>>>     <webAppContext contextPath="/admin"
>>> resourceBase="${activemq.base}/webapps/admin" logUrlOnStart="true"/>
>>>     <webAppContext contextPath="/demo"
>>> resourceBase="${activemq.base}/webapps/demo" logUrlOnStart="true"/>
>>>     <webAppContext contextPath="/fileserver"
>>> resourceBase="${activemq.base}/webapps/fileserver"
>>> logUrlOnStart="true"/>
>>>   </handlers>
>>> </jetty>
>>>
>>> <!-- Embedded Derby DataSource Sample Setup -->
>>> <bean id="derby-ds"  
>>> class="org.apache.derby.jdbc.EmbeddedDataSource">
>>>   <property name="databaseName" value="derbydb"/>
>>>   <property name="createDatabase" value="create"/>
>>> </bean>
>>>
>>> </beans>
>>> <!-- END SNIPPET: example -->
>>>
>>>
>>>
>>> ----------------------
>>>
>>>
>>> And our Java clients use the following URL to connect to the server:
>>>
>>> tcp://${jms.server}:${jms.port}?
>>> keepAlive
>>> =
>>> true
>>> &amp
>>> ;connectionTimeout
>>> =
>>> 0
>>> &amp
>>> ;wireFormat
>>> .maxInactivityDuration
>>> =0&amp;jms.useAsyncSend=true&amp;jms.prefetchPolicy.all=250"
>>>
>>>
>>> Notice that I tried setting the connectionTimeout and
>>> maxInactivityDuration
>>> to "0" thinking that this would turn off inactivity checking.
>>> This didn't help.  Further, the system does try to send through a
>>> LOT of
>>> traffic when we bring it up, so there's definitely no "inactivity".
>>>
>>> As I mentioned, the exact same codebase was working perfectly fine
>>> with
>>> ActiveMQ 4.1.1.  There's been no code changes, and no changes in the
>>> nature
>>> of the logs we're sending through.  The only change has been
>>> upgrading to
>>> AMQ 5.2.0 on the server side, and using the new jar file on the
>>> Java client side.
>>>
>>> Any help would be appreciated, as I'm totally stumped as to why the
>>> server
>>> keeps restarting itself.  Our message content hasn't changed,
>>> so why would the EOF file errors or "channel inactive" errors be
>>> happening??
>>>
>>> Thanks,
>>> Gary
>>>
>>> -- 
>>> View this message in context:
>>> http://www.nabble.com/Standalone-server-restarting-itself-every-5-minutes---java.lang.IllegalStateException-tp24727769p24727769.html
>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>
>>
>> Rob Davies
>> I work here: http://fusesource.com
>> My Blog: http://rajdavies.blogspot.com/
>> I'm writing this: http://www.manning.com/snyder/
>>
>>
>>
>>
>>
>>
>
> -- 
> View this message in context: http://www.nabble.com/Standalone-server-restarting-itself-every-5-minutes---java.lang.IllegalStateException-tp24727769p24743651.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>

Rob Davies
I work here: http://fusesource.com
My Blog: http://rajdavies.blogspot.com/
I'm writing this: http://www.manning.com/snyder/





Mime
View raw message