stratos-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Martin Eppel (meppel)" <mep...@cisco.com>
Subject RE: [grouping] [question] stratos error during startup
Date Sat, 01 Nov 2014 23:37:21 GMT
I think you guys are right,

It seems I was able to isolate the issue and it appears to be related to the crm which we
use to start, stop and monitor processes. Removing the activemq and stratos from the crm config
and instead invoke it simply through a shell script as part of the octl startup process fixed
(at least temporarily as a workaround) the issue.

Actually, shutdown during tenant creation was reported by Jeffrey, I typically don’t see
this in my environment

Thanks

Martin

From: Udara Liyanage [mailto:udara@wso2.com]
Sent: Friday, October 31, 2014 9:38 PM
To: dev
Subject: Re: [grouping] [question] stratos error during startup


Hi Martin,

You have reported shutdown thing when tenant creation also. It seems another process kill/send
SIGINT to Stratos. May be it tries to kill when Stratos produces error messages.



Touched, not typed. Erroneous words are a feature, not a typo.
On Oct 31, 2014 11:42 PM, "Nirmal Fernando" <nirmal070125@gmail.com<mailto:nirmal070125@gmail.com>>
wrote:
Hi Martin,

Is this a puppet based deployment? If so please check whether it's puppet causing this.

On Fri, Oct 31, 2014 at 6:31 PM, Martin Eppel (meppel) <meppel@cisco.com<mailto:meppel@cisco.com>>
wrote:
Yes, I tried telnet (telnet octl.qmog.cisco.com<http://octl.qmog.cisco.com> 61616) after
the exception happened and it is accessible.

According to the stratos log it will retry to connect in 30 seconds but then the shutdown
hook is triggered. My question why / what will trigger the shutdown (see logs below) ?

TID: [0] [STRATOS] [2014-10-31 17:22:27,196] ERROR {org.apache.stratos.messaging.broker.subscribe.TopicSubscriber}
-  Error while subscribing to the topic: application-status {org.apache.stratos.messaging.broker.subscribe.TopicSubscriber}
javax.jms.JMSException: Could not connect to broker URL: tcp://octl.qmog.cisco.com:61616<http://octl.qmog.cisco.com:61616>.
Reason: java.net.ConnectException: Connection refused
        at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:36)
       at org.apache.activemq.ActiveMQConnectionFactory.createActiveMQConnection(ActiveMQConnectionFactory.java:358)
        at org.apache.activemq.ActiveMQConnectionFactory.createActiveMQConnection(ActiveMQConnectionFactory.java:303)
        at org.apache.activemq.ActiveMQConnectionFactory.createTopicConnection(ActiveMQConnectionFactory.java:277)
        at org.apache.stratos.messaging.broker.connect.TopicConnector.init(TopicConnector.java:68)
        at org.apache.stratos.messaging.broker.subscribe.TopicSubscriber.doSubscribe(TopicSubscriber.java:64)
        at org.apache.stratos.messaging.broker.subscribe.TopicSubscriber.run(TopicSubscriber.java:117)
       at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.ConnectException: Connection refused
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
       at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)        at java.net.Socket.connect(Socket.java:579)
        at org.apache.activemq.transport.tcp.TcpTransport.connect(TcpTransport.java:501) 
      at org.apache.activemq.transport.tcp.TcpTransport.doStart(TcpTransport.java:464)
        at org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)
        at org.apache.activemq.transport.AbstractInactivityMonitor.start(AbstractInactivityMonitor.java:138)
        at org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:58)
        at org.apache.activemq.transport.WireFormatNegotiator.start(WireFormatNegotiator.java:72)
       at org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:58)
        at org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:58)  
     at org.apache.activemq.ActiveMQConnectionFactory.createActiveMQConnection(ActiveMQConnectionFactory.java:338)
        ... 6 moreTID: [0] [STRATOS] [2014-10-31 17:22:27,199]  INFO {org.apache.stratos.messaging.broker.subscribe.TopicSubscriber}
-  Will try to subscribe again in 30 sec {org.apache.stratos.messaging.broker.subscribe.TopicSubscriber}TID:
[0] [STRATOS] [2014-10-31 17:22:27,520]  INFO {org.wso2.carbon.core.init.CarbonServerManager}
-  Shutdown hook triggered.... {org.wso2.carbon.core.init.CarbonServerManager}TID: [0] [STRATOS]
[2014-10-31 17:22:27,524]  INFO {org.wso2.carbon.core.init.CarbonServerManager} -  Gracefully
shutting down Apache Stratos... {org.wso2.carbon.core.init.CarbonServerManager}TID: [0] [STRATOS]
[2014-10-31 17:22:27,531]  INFO {org.wso2.carbon.core.ServerManagement} -  Starting to switch
to maintenance mode... {org.wso2.carbon.core.ServerManagement}TID: [0] [STRATOS] [2014-10-31
17:22:27,532]  INFO {org.wso2.carbon.core.ServerManagement} -  Stopped all transport listeners
{org.wso2.carbon.core.ServerManagement}TID: [0] [STRATOS] [2014-10-31 17:22:27,532]  INFO
{org.wso2.carbon.core.ServerManagement} -  Waiting for request service completion... {org.wso2.carbon.core.ServerManagement}

From: Martin Eppel (meppel)
Sent: Friday, October 31, 2014 10:02 AM
To: dev@stratos.apache.org<mailto:dev@stratos.apache.org>
Subject: RE: [grouping] [question] stratos error during startup

Ok, I try it.

One questions, does stratos limits the numbers of retries or does it eventually shut down
(this is what it seems to happen that stratos shuts down after it fails to connect to the
activemq)

Thanks

Martin

From: Imesh Gunaratne [mailto:imesh@apache.org]
Sent: Friday, October 31, 2014 9:57 AM
To: dev
Subject: Re: [grouping] [question] stratos error during startup

Hi Martin,

According to the error log it seems like Stratos could not reach the message broker at the
given tcp URL. However it should try to reconnect afterwards.

Will us be able to check whether it is possible to telnet to this tcp socket fromt he Stratos
host?

Thanks

On Fri, Oct 31, 2014 at 7:49 PM, Martin Eppel (meppel) <meppel@cisco.com<mailto:meppel@cisco.com>>
wrote:
Yes it is running when stratos starts up

From: Udara Liyanage [mailto:udara@wso2.com<mailto:udara@wso2.com>]
Sent: Thursday, October 30, 2014 8:09 PM
To: dev
Subject: Re: [grouping] [question] stratos error during startup


Hi Martin,

Is ActiveMQ available at the time of Stratos startup time?

Touched, not typed. Erroneous words are a feature, not a typo.
On Oct 31, 2014 2:10 AM, "Martin Eppel (meppel)" <meppel@cisco.com<mailto:meppel@cisco.com>>
wrote:
I am trying to migrate the grouping stratos binaries to our orchestration environment but
consistently run into the issue that when stratos starts up the first time an (activemq) eror
is thrown when trying to subscribe to application-status (Error while subscribing to the topic:
application-status). ActiveMq seems to be up and running and after restarting stratos,  stratos
comes up without the exception (without having activemq to restart).

Any hints / ideas what could cause this or what I should look for ?

Thanks

Martin

(I attached the complete log)


TID: [0] [STRATOS] [2014-10-30 19:34:52,071]  INFO {org.wso2.carbon.ui.internal.CarbonUIServiceComponent}
-  Mgt Console URL  : https://172.16.2.207:9443/carbon/ {org.wso2.carbon.ui.internal.CarbonUIServiceComponent}
TID: [0] [STRATOS] [2014-10-30 19:34:58,071]  INFO {org.apache.stratos.cep.extension.CEPTopologyEventReceiver}
-  CEP topology receiver thread started {org.apache.stratos.cep.extension.CEPTopologyEventReceiver}
TID: [0] [STRATOS] [2014-10-30 19:34:58,072]  INFO {org.apache.stratos.messaging.message.receiver.topology.TopologyEventMessageDelegator}
-  Topology event message delegator started {org.apache.stratos.messaging.message.receiver.topology.TopologyEventMessageDelegator}
TID: [0] [STRATOS] [2014-10-30 19:35:01,024] DEBUG {org.apache.stratos.cloud.controller.topology.TopologySynchronizerTask}
-  Executing topology synchronization task {org.apache.stratos.cloud.controller.topology.TopologySynchronizerTask}
TID: [0] [STRATOS] [2014-10-30 19:35:01,025] DEBUG {org.apache.stratos.cloud.controller.topology.TopologyManager}
-  Trying to retrieve topology from registry {org.apache.stratos.cloud.controller.topology.TopologyManager}
TID: [0] [STRATOS] [2014-10-30 19:35:01,027] DEBUG {org.apache.stratos.manager.publisher.TenantSynzhronizerTask}
-  Publishing complete tenant event {org.apache.stratos.manager.publisher.TenantSynzhronizerTask}
TID: [0] [STRATOS] [2014-10-30 19:35:01,032] DEBUG {org.apache.stratos.cloud.controller.topology.TopologyManager}
-  Topology not found in registry, creating new {org.apache.stratos.cloud.controller.topology.TopologyManager}
TID: [0] [STRATOS] [2014-10-30 19:35:01,033] DEBUG {org.apache.stratos.cloud.controller.topology.TopologyManager}
-  Topology initialized {org.apache.stratos.cloud.controller.topology.TopologyManager}
TID: [0] [STRATOS] [2014-10-30 19:35:01,035] DEBUG {org.apache.stratos.cloud.controller.topology.TopologyEventPublisher}
-  Publishing complete topology event {org.apache.stratos.cloud.controller.topology.TopologyEventPublisher}
TID: [0] [STRATOS] [2014-10-30 19:35:01,155]  INFO {org.apache.stratos.messaging.message.processor.topology.CompleteTopologyMessageProcessor}
-  Topology initialized {org.apache.stratos.messaging.message.processor.topology.CompleteTopologyMessageProcessor}
TID: [0] [STRATOS] [2014-10-30 19:35:01,156]  INFO {org.apache.stratos.messaging.message.processor.topology.CompleteTopologyMessageProcessor}
-  Topology initialized {org.apache.stratos.messaging.message.processor.topology.CompleteTopologyMessageProcessor}
TID: [0] [STRATOS] [2014-10-30 19:35:01,156]  INFO {org.apache.stratos.messaging.message.processor.topology.CompleteTopologyMessageProcessor}
-  Topology initialized {org.apache.stratos.messaging.message.processor.topology.CompleteTopologyMessageProcessor}
TID: [0] [STRATOS] [2014-10-30 19:35:01,158]  INFO {org.apache.stratos.autoscaler.message.receiver.topology.AutoscalerTopologyEventReceiver}
-  [CompleteTopologyEvent] Received: class org.apache.stratos.messaging.event.topology.CompleteTopologyEvent
{org.apache.stratos.autoscaler.message.receiver.topology.AutoscalerTopologyEventReceiver}
TID: [0] [STRATOS] [2014-10-30 19:35:01,158]  INFO {org.apache.stratos.messaging.message.processor.topology.CompleteTopologyMessageProcessor}
-  Topology initialized {org.apache.stratos.messaging.message.processor.topology.CompleteTopologyMessageProcessor}
TID: [0] [STRATOS] [2014-10-30 19:35:01,159]  INFO {org.apache.stratos.manager.topology.receiver.StratosManagerTopologyEventReceiver}
-  [CompleteTopologyEventListener] Received: class org.apache.stratos.messaging.event.topology.CompleteTopologyEvent
{org.apache.stratos.manager.topology.receiver.StratosManagerTopologyEventReceiver}
TID: [0] [STRATOS] [2014-10-30 19:35:01,160]  INFO {org.apache.stratos.cep.extension.CEPTopologyEventReceiver}
-  Complete topology event received to fault handling window processor. {org.apache.stratos.cep.extension.CEPTopologyEventReceiver}
TID: [0] [STRATOS] [2014-10-30 19:35:01,160]  INFO {org.apache.stratos.cep.extension.FaultHandlingWindowProcessor}
-  Member time stamp map was successfully loaded from the topology. {org.apache.stratos.cep.extension.FaultHandlingWindowProcessor}
TID: [0] [STRATOS] [2014-10-30 19:35:04,168] DEBUG {org.apache.stratos.manager.publisher.TenantEventPublisher}
-  Publishing tenant created event: [tenant-id] 1 [tenant-domain] cloud1.qmog.cisco.com<http://cloud1.qmog.cisco.com>
{org.apache.stratos.manager.publisher.TenantEventPublisher}
TID: [0] [STRATOS] [2014-10-30 19:35:06,045] ERROR {org.apache.stratos.messaging.broker.heartbeat.TopicHealthChecker}
-  instance-status topic health checker is failed and will try to subscribe again in 30 seconds.
{org.apache.stratos.messaging.broker.heartbeat.TopicHealthChecker}
TID: [0] [STRATOS] [2014-10-30 19:35:06,104] ERROR {org.apache.stratos.messaging.broker.heartbeat.TopicHealthChecker}
-  instance-status topic health checker is failed and will try to subscribe again in 30 seconds.
{org.apache.stratos.messaging.broker.heartbeat.TopicHealthChecker}
TID: [0] [STRATOS] [2014-10-30 19:35:06,106] ERROR {org.apache.stratos.messaging.broker.heartbeat.TopicHealthChecker}
-  topology topic health checker is failed and will try to subscribe again in 30 seconds.
{org.apache.stratos.messaging.broker.heartbeat.TopicHealthChecker}
TID: [0] [STRATOS] [2014-10-30 19:35:06,225] ERROR {org.apache.stratos.messaging.broker.heartbeat.TopicHealthChecker}
-  topology topic health checker is failed and will try to subscribe again in 30 seconds.
{org.apache.stratos.messaging.broker.heartbeat.TopicHealthChecker}
TID: [0] [STRATOS] [2014-10-30 19:35:06,229] ERROR {org.apache.stratos.messaging.broker.heartbeat.TopicHealthChecker}
-  topology topic health checker is failed and will try to subscribe again in 30 seconds.
{org.apache.stratos.messaging.broker.heartbeat.TopicHealthChecker}
TID: [0] [STRATOS] [2014-10-30 19:35:06,319] ERROR {org.apache.stratos.messaging.broker.heartbeat.TopicHealthChecker}
-  summarized-health-stats topic health checker is failed and will try to subscribe again
in 30 seconds. {org.apache.stratos.messaging.broker.heartbeat.TopicHealthChecker}
TID: [0] [STRATOS] [2014-10-30 19:35:06,397] ERROR {org.apache.stratos.messaging.broker.heartbeat.TopicHealthChecker}
-  topology topic health checker is failed and will try to subscribe again in 30 seconds.
{org.apache.stratos.messaging.broker.heartbeat.TopicHealthChecker}
TID: [0] [STRATOS] [2014-10-30 19:35:06,656]  INFO {org.apache.stratos.cloud.controller.application.status.receiver.ApplicationStatusTopicReceiver}
-  Cloud controller application status thread started {org.apache.stratos.cloud.controller.application.status.receiver.ApplicationStatusTopicReceiver}
TID: [0] [STRATOS] [2014-10-30 19:35:06,658]  INFO {org.apache.stratos.messaging.message.receiver.application.status.ApplicationStatusEventMessageDelegator}
-  Application status event message delegator started {org.apache.stratos.messaging.message.receiver.application.status.ApplicationStatusEventMessageDelegator}
TID: [0] [STRATOS] [2014-10-30 19:35:06,663] ERROR {org.apache.stratos.messaging.broker.subscribe.TopicSubscriber}
-  Error while subscribing to the topic: application-status {org.apache.stratos.messaging.broker.subscribe.TopicSubscriber}
javax.jms.JMSException: Could not connect to broker URL: tcp://octl.qmog.cisco.com:61616<http://octl.qmog.cisco.com:61616>.
Reason: java.net.ConnectException: Connection refused
        at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:36)
        at org.apache.activemq.ActiveMQConnectionFactory.createActiveMQConnection(ActiveMQConnectionFactory.java:358)
        at org.apache.activemq.ActiveMQConnectionFactory.createActiveMQConnection(ActiveMQConnectionFactory.java:303)
        at org.apache.activemq.ActiveMQConnectionFactory.createTopicConnection(ActiveMQConnectionFactory.java:277)
        at org.apache.stratos.messaging.broker.connect.TopicConnector.init(TopicConnector.java:68)
        at org.apache.stratos.messaging.broker.subscribe.TopicSubscriber.doSubscribe(TopicSubscriber.java:64)
        at org.apache.stratos.messaging.broker.subscribe.TopicSubscriber.run(TopicSubscriber.java:117)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.ConnectException: Connection refused
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:579)
        at org.apache.activemq.transport.tcp.TcpTransport.connect(TcpTransport.java:501)
        at org.apache.activemq.transport.tcp.TcpTransport.doStart(TcpTransport.java:464)
        at org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)
        at org.apache.activemq.transport.AbstractInactivityMonitor.start(AbstractInactivityMonitor.java:138)
        at org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:58)
        at org.apache.activemq.transport.WireFormatNegotiator.start(WireFormatNegotiator.java:72)
        at org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:58)
        at org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:58)
        at org.apache.activemq.ActiveMQConnectionFactory.createActiveMQConnection(ActiveMQConnectionFactory.java:338)
        ... 6 more
TID: [0] [STRATOS] [2014-10-30 19:35:06,666]  INFO {org.apache.stratos.messaging.broker.subscribe.TopicSubscriber}
-  Will try to subscribe again in 30 sec {org.apache.stratos.messaging.broker.subscribe.TopicSubscriber}
TID: [0] [STRATOS] [2014-10-30 19:35:07,146]  INFO {org.wso2.carbon.core.init.CarbonServerManager}
-  Shutdown hook triggered.... {org.wso2.carbon.core.init.CarbonServerManager}
TID: [0] [STRATOS] [2014-10-30 19:35:07,152]  INFO {org.wso2.carbon.core.init.CarbonServerManager}
-  Gracefully shutting down Apache Stratos... {org.wso2.carbon.core.init.CarbonServerManager}
TID: [0] [STRATOS] [2014-10-30 19:35:07,156]  INFO {org.wso2.carbon.core.ServerManagement}
-  Starting to switch to maintenance mode... {org.wso2.carbon.core.ServerManagement}
TID: [0] [STRATOS] [2014-10-30 19:35:07,158]  INFO {org.wso2.carbon.core.ServerManagement}
-  Stopped all transport listeners {org.wso2.carbon.core.ServerManagement}
TID: [0] [STRATOS] [2014-10-30 19:35:07,158]  INFO {org.wso2.carbon.core.ServerManagement}
-  Waiting for request service completion... {org.wso2.carbon.core.ServerManagement}
TID: [0] [STRATOS] [2014-10-30 19:35:07,162]  INFO {org.wso2.carbon.core.ServerManagement}
-  All requests have been served. {org.wso2.carbon.core.ServerManagement}
TID: [0] [STRATOS] [2014-10-30 19:35:07,162]  INFO {org.wso2.carbon.core.ServerManagement}
-  Waiting for deployment completion... {org.wso2.carbon.core.ServerManagement}
TID: [0] [STRATOS] [2014-10-30 19:35:07,180]  INFO {org.wso2.carbon.webapp.mgt.WebApplication}
-  Unloaded webapp: StandardEngine[Catalina].StandardHost[localhost].StandardContext[/STRATOS_ROOT]
{org.wso2.carbon.webapp.mgt.WebApplication}
TID: [0] [STRATOS] [2014-10-30 19:35:07,181]  INFO {org.springframework.web.context.support.XmlWebApplicationContext}



--
Imesh Gunaratne

Technical Lead, WSO2
Committer & PMC Member, Apache Stratos



--
Best Regards,
Nirmal

Nirmal Fernando.
PPMC Member & Committer of Apache Stratos,
Senior Software Engineer, WSO2 Inc.

Blog: http://nirmalfdo.blogspot.com/
Mime
View raw message