Mailing-List: contact users-help@activemq.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@activemq.apache.org
Received-SPF: pass (nike.apache.org: domain of rajdavies@gmail.com designates
 209.85.217.207 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:from:to:in-reply-to:content-type
         :content-transfer-encoding:mime-version:subject:date:references
         :x-mailer;
        b=em7yk/xUlfuTZt4aHKJgJ0hX0uXjErQdf8SCu9TKHBlcSJVPj+6vd1XUGrkxnlZEzC
         JK6b11sOE3TKkWWr6TLbKgZtuykT+4IMX0zcp74a8JCMXlpoPxmotrrg6nglBcu1PhMx
         BeX644SoGG2FHTybWxDFCkZ0yeMpDY8tXR6o4=
Message-Id: <B1B5BD69-7BC3-4103-8C1A-36C769C5F530@gmail.com>
From: Rob Davies <rajdavies@gmail.com>
To: users@activemq.apache.org
In-Reply-To: <24727769.post@talk.nabble.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v935.3)
Subject: Re: Standalone server restarting itself every 5 minutes -
 java.lang.IllegalStateException
Date: Thu, 30 Jul 2009 12:34:34 +0100
References: <24727769.post@talk.nabble.com>

first - try disabling multicast discovery in the activemq configuration

On 29 Jul 2009, at 22:17, GaryG wrote:

>
> We've recently updated to ActiveMQ 5.2.0, and are running it as a  
> standalone
> server, with a config file very similar to the basic example  
> provided out of
> the box.  There has been no code change in the Java code that talks  
> to the
> AMQ server.  Yet, once started up, the ActiveMQ server process is  
> restarting
> itself every 5 minutes.
>
> Looking at the "activemq.log", I see the following:
> ===============
> .
> .
> .
> 2009-07-29 20:48:01,949 [31.140.17:41258] DEBUG AbstractRegion
> - Adding destination: topic:// 
> ActiveMQ.Advisory.Producer.Topic.dr01b_wf1
> 2009-07-29 20:48:01,951 [31.140.17:41258] DEBUG AbstractRegion
> - Adding destination:
> topic://ActiveMQ.Advisory.Producer.Topic.WorkerFarmControl
> 2009-07-29 20:48:01,952 [31.140.17:41258] DEBUG AbstractRegion
> - Adding destination: topic:// 
> ActiveMQ.Advisory.Producer.Topic.dr01a_wf1
> 2009-07-29 20:48:01,956 [31.140.17:41258] DEBUG AbstractRegion
> - Adding destination: topic://ControllerCCTopicSend
> 2009-07-29 20:48:01,957 [31.140.17:41258] DEBUG  
> JournalPersistenceAdapter
> - Waking for checkpoint to complete.
> 2009-07-29 20:48:01,957 [eckpoint Worker] DEBUG  
> JournalPersistenceAdapter
> - Checkpoint started.
> 2009-07-29 20:48:01,979 [eckpoint Worker] DEBUG  
> JournalPersistenceAdapter
> - Checkpoint done.
> 2009-07-29 20:48:01,984 [31.140.17:41258] DEBUG AbstractRegion
> - Adding destination:
> topic://ActiveMQ.Advisory.Producer.Topic.ControllerCCTopicSend
> 2009-07-29 20:52:55,790 [MQ ShutdownHook] INFO  BrokerService
> - ActiveMQ Message Broker (ai_jms,  
> ID:server2-55452-1248900479410-0:0) is
> shutting down
> 2009-07-29 20:52:55,791 [MQ ShutdownHook] DEBUG BrokerService
> - Caught exception, must be shutting down:  
> java.lang.IllegalStateException:
> Shutdown in progress
> 2009-07-29 20:52:55,796 [MQ ShutdownHook] INFO  NetworkConnector
> - Network Connector local-nc Stopped
> 2009-07-29 20:52:57,294 [31.140.18:56300] DEBUG TransportConnection
> - Stopping connection: /1.1.1.18:56300
> 2009-07-29 20:52:57,294 [31.140.18:56300] DEBUG TcpTransport
> - Stopping transport tcp:///1.1.1.18:56300
> 2009-07-29 20:52:57,297 [31.140.18:56300] DEBUG TransportConnection
> - Stopped transport: /1.1.1.18:56300
> 2009-07-29 20:52:57,298 [31.140.18:56300] DEBUG TransportConnection
> - Cleaning up connection resources: /1.1.1.18:56300
> 2009-07-29 20:52:57,299 [31.140.18:56300] DEBUG  
> JournalPersistenceAdapter
> - Waking for checkpoint to complete.
> 2009-07-29 20:52:57,299 [eckpoint Worker] DEBUG  
> JournalPersistenceAdapter
> - Checkpoint started.
> 2009-07-29 20:52:57,331 [eckpoint Worker] DEBUG  
> JournalPersistenceAdapter
> - Checkpoint done.
> 2009-07-29 20:52:57,338 [31.140.18:56300] DEBUG AbstractRegion
> - Removing consumer: ID:server2-49559-1248899931358-0:2:-1:1
> .
> .
> .
> -------------
>
> So you can see for a while, the broker's adding topics and such, and  
> then
> all of a sudden it gets an "IllegalStateException", and just restarts.
>
>
> On the clients, when this happens we get this error:
>
> --------------
> 2009-07-29 20:52:48,550 [ActiveMQ Session Task] DEBUG jms_comm -
> ActiveMQMessageReceiver[dr01b_wf1] got msg from = STG_AIC1,
> jmsMsgID=ID:server1-41646-1248899975358-0:2:6:1:224
> 2009-07-29 20:52:48,550 [ActiveMQ Session Task] DEBUG jms_comm -
> ActiveMQMessageReceiver[dr01b_wf1] sent processed msg to  
> messageListener!
> 2009-07-29 20:52:49,896 [ActiveMQ Session Task] DEBUG jms_comm -
> ActiveMQMessageReceiver[dr01b_wf1] got msg from = STG_AIC1,
> jmsMsgID=ID:server1-41646-1248899975358-0:2:6:1:225
> 2009-07-29 20:52:49,896 [ActiveMQ Session Task] DEBUG jms_comm -
> ActiveMQMessageReceiver[dr01b_wf1] sent processed msg to  
> messageListener!
> 2009-07-29 20:52:57,865 [ActiveMQ Connection Worker:
> tcp://server2/1.1.1.18:61616] ERROR jms_comm - FactoryUtil.onException
> javax.jms.JMSException: java.io.EOFException
>        at
> org 
> .apache 
> .activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:49)
>        at
> org 
> .apache 
> .activemq 
> .ActiveMQConnection.onAsyncException(ActiveMQConnection.java:1773)
>        at
> org 
> .apache 
> .activemq.ActiveMQConnection.onException(ActiveMQConnection.java:1790)
>        at
> org 
> .apache 
> .activemq.transport.TransportFilter.onException(TransportFilter.java: 
> 99)
>        at
> org 
> .apache 
> .activemq 
> .transport.ResponseCorrelator.onException(ResponseCorrelator.java:114)
>        at
> org 
> .apache 
> .activemq.transport.TransportFilter.onException(TransportFilter.java: 
> 99)
>        at
> org 
> .apache 
> .activemq.transport.TransportFilter.onException(TransportFilter.java: 
> 99)
>        at
> org 
> .apache 
> .activemq 
> .transport 
> .WireFormatNegotiator.onException(WireFormatNegotiator.java:152)
>        at
> org 
> .apache 
> .activemq 
> .transport.InactivityMonitor.onException(InactivityMonitor.java:244)
>        at
> org 
> .apache 
> .activemq 
> .transport.TransportSupport.onException(TransportSupport.java:96)
>        at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java: 
> 189)
>        at java.lang.Thread.run(Thread.java:619)
> Caused by: java.io.EOFException
>        at java.io.DataInputStream.readInt(DataInputStream.java:375)
>        at
> org 
> .apache 
> .activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
>        at
> org 
> .apache 
> .activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java: 
> 210)
>        at
> org 
> .apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java: 
> 202)
>        at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java: 
> 185)
>        ... 1 more
> 2009-07-29 20:52:57,895 [ActiveMQ Connection Worker:
> tcp://server2.mss.iss.net/1.1.1.18:61616] DEBUG jms_comm - JMS  
> Exception
> occured. Server is down.
> 2009-07-29 20:52:58,263 [Thread-27] ERROR jms_comm.log -
> ActiveMQMessageSender[ControllerTaskStatusTopic] Error sending  
> message:
> javax.jms.JMSException: Channel was inactive for too long:
> server2/1.1.1.18:61616
> --------------
>
>
> Our "activemq.xml" file is pretty simple - it's almost exactly the  
> default
> config with just some minor modifications:
> ----------------------
>
> <!--
>    Licensed to the Apache Software Foundation (ASF) under one or more
>    contributor license agreements.  See the NOTICE file distributed  
> with
>    this work for additional information regarding copyright ownership.
>    The ASF licenses this file to You under the Apache License,  
> Version 2.0
>    (the "License"); you may not use this file except in compliance  
> with
>    the License.  You may obtain a copy of the License at
>
>    http://www.apache.org/licenses/LICENSE-2.0
>
>    Unless required by applicable law or agreed to in writing, software
>    distributed under the License is distributed on an "AS IS" BASIS,
>    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or  
> implied.
>    See the License for the specific language governing permissions and
>    limitations under the License.
> -->
> <!-- START SNIPPET: example -->
> <beans
>  xmlns="http://www.springframework.org/schema/beans"
>  xmlns:amq="http://activemq.apache.org/schema/core"
>  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>  xsi:schemaLocation="http://www.springframework.org/schema/beans
> http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
>  http://activemq.apache.org/schema/core
> http://activemq.apache.org/schema/core/activemq-core.xsd
>  http://activemq.apache.org/camel/schema/spring
> http://activemq.apache.org/camel/schema/spring/camel-spring.xsd">
>
>  <!-- Allows us to use system properties as variables in this  
> configuration
> file -->
>  <bean
> class 
> = 
> "org 
> .springframework.beans.factory.config.PropertyPlaceholderConfigurer">
>    <property name="locations">
>      <value>file:///${activemq.base}/conf/credentials.properties</ 
> value>
>    </property>
>  </bean>
>
>  <broker xmlns="http://activemq.apache.org/schema/core"  
> brokerName="ai_jms"
> useJmx="true" dataDirectory="${activemq.base}/data">
>
>    <!-- The store and forward broker networks ActiveMQ will listen  
> to -->
>    <networkConnectors>
>      <!-- by default just auto discover the other brokers -->
>      <!--<networkConnector name="default-nc" uri="multicast:// 
> default"/>
> -->
>      <!-- Example of a static configuration: -->
>      <networkConnector name="local-nc"
> uri="static://(tcp://localhost:61616)"/>
>    </networkConnectors>
>
>    <persistenceAdapter>
>      <journaledJDBC journalLogFiles="10"
> dataDirectory="${activemq.base}/activemq-data" dataSource="#derby- 
> ds"/>
>    </persistenceAdapter>
>
>
>    <!--  The maximum about of space the broker will use before  
> slowing down
> producers -->
>    <systemUsage>
>      <systemUsage>
>        <memoryUsage>
>          <memoryUsage limit="512 mb"/>
>        </memoryUsage>
>        <storeUsage>
>          <storeUsage limit="1 gb" name="foo"/>
>        </storeUsage>
>        <tempUsage>
>          <tempUsage limit="100 mb"/>
>        </tempUsage>
>      </systemUsage>
>    </systemUsage>
>
>    <!-- The transport connectors ActiveMQ will listen to -->
>    <transportConnectors>
>      <transportConnector name="openwire" uri="tcp://localhost:61616"
> discoveryUri="multicast://default"/>
>    </transportConnectors>
>
>  </broker>
>
>  <!-- An embedded servlet engine for serving up the Admin console -->
>  <jetty xmlns="http://mortbay.com/schemas/jetty/1.0">
>    <connectors>
>      <nioConnector port="8161"/>
>    </connectors>
>
>    <handlers>
>      <webAppContext contextPath="/admin"
> resourceBase="${activemq.base}/webapps/admin" logUrlOnStart="true"/>
>      <webAppContext contextPath="/demo"
> resourceBase="${activemq.base}/webapps/demo" logUrlOnStart="true"/>
>      <webAppContext contextPath="/fileserver"
> resourceBase="${activemq.base}/webapps/fileserver"  
> logUrlOnStart="true"/>
>    </handlers>
>  </jetty>
>
>  <!-- Embedded Derby DataSource Sample Setup -->
>  <bean id="derby-ds" class="org.apache.derby.jdbc.EmbeddedDataSource">
>    <property name="databaseName" value="derbydb"/>
>    <property name="createDatabase" value="create"/>
>  </bean>
>
> </beans>
>  <!-- END SNIPPET: example -->
>
>
>
> ----------------------
>
>
> And our Java clients use the following URL to connect to the server:
>
> tcp://${jms.server}:${jms.port}? 
> keepAlive 
> = 
> true 
> &amp 
> ;connectionTimeout 
> = 
> 0 
> &amp 
> ;wireFormat 
> .maxInactivityDuration 
> =0&amp;jms.useAsyncSend=true&amp;jms.prefetchPolicy.all=250"
>
>
> Notice that I tried setting the connectionTimeout and  
> maxInactivityDuration
> to "0" thinking that this would turn off inactivity checking.
> This didn't help.  Further, the system does try to send through a  
> LOT of
> traffic when we bring it up, so there's definitely no "inactivity".
>
> As I mentioned, the exact same codebase was working perfectly fine  
> with
> ActiveMQ 4.1.1.  There's been no code changes, and no changes in the  
> nature
> of the logs we're sending through.  The only change has been  
> upgrading to
> AMQ 5.2.0 on the server side, and using the new jar file on the
> Java client side.
>
> Any help would be appreciated, as I'm totally stumped as to why the  
> server
> keeps restarting itself.  Our message content hasn't changed,
> so why would the EOF file errors or "channel inactive" errors be  
> happening??
>
> Thanks,
> Gary
>
> -- 
> View this message in context: http://www.nabble.com/Standalone-server-restarting-itself-every-5-minutes---java.lang.IllegalStateException-tp24727769p24727769.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>

Rob Davies
I work here: http://fusesource.com
My Blog: http://rajdavies.blogspot.com/
I'm writing this: http://www.manning.com/snyder/