stratos-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Martin Eppel (meppel)" <mep...@cisco.com>
Subject RE: Testing Stratos 4.1: Application undeployment: application fails to undeploy (nested grouping, group scaling)
Date Fri, 05 Jun 2015 17:03:50 GMT
Ok:

log4j.logger.org.apache.stratos.manager=DEBUG
log4j.logger.org.apache.stratos.autoscaler=DEBUG
log4j.logger.org.apache.stratos.messaging=INFO
log4j.logger.org.apache.stratos.cloud.controller=DEBUG
log4j.logger.org.wso2.andes.client=ERROR
# Autoscaler rule logs
log4j.logger.org.apache.stratos.autoscaler.rule.RuleLog=DEBUG

From: Udara Liyanage [mailto:udara@wso2.com]
Sent: Friday, June 05, 2015 10:00 AM
To: dev
Subject: Re: Testing Stratos 4.1: Application undeployment: application fails to undeploy
(nested grouping, group scaling)

Hi Martin,

Better if you can enable debugs logs for all AS, CC and cartridge agent

On Fri, Jun 5, 2015 at 10:23 PM, Udara Liyanage <udara@wso2.com<mailto:udara@wso2.com>>
wrote:
Hi,

Please enable AS debug logs.

On Fri, Jun 5, 2015 at 9:38 PM, Martin Eppel (meppel) <meppel@cisco.com<mailto:meppel@cisco.com>>
wrote:
Hi Udara,

Yes, this issue seems to be fairly well reproducible, which debug log do you want me to enable,
cartridge agent logs ?

Thanks

Martin

From: Udara Liyanage [mailto:udara@wso2.com<mailto:udara@wso2.com>]
Sent: Thursday, June 04, 2015 11:11 PM
To: dev
Subject: Re: Testing Stratos 4.1: Application undeployment: application fails to undeploy
(nested grouping, group scaling)

Hi,

This might be possible if AS did not receive member activated event published by CC. Is it
possible to enable debug logs if this is reproducible.
Or else I can add an INFO logs and commit.


On Fri, Jun 5, 2015 at 9:11 AM, Udara Liyanage <udara@wso2.com<mailto:udara@wso2.com>>
wrote:
Hi,


For the first issue you have mentioned, the particular member is activated, but it is still
identified as an obsolete member and is being marked to be terminated since pending time expired.
Does that mean member is still in Obsolete list even though it is being activated?

//member started
TID: [0] [STRATOS] [2015-06-04 19:53:04,706]  INFO {org.apache.stratos.autoscaler.context.cluster.ClusterContext}
-  Member stat context has been added: [application] g-sc-G12-1 [cluster] g-sc-G12-1.c1-0x0.c1.domain
[clusterInstanceContext] g-sc-G12-1-1 [partitionContext] whole-region [member-id] g-sc-G12-1.c1-0x0.c1.domainb0aa0188-49f1-47f6-a040-c2eab4acb5b1

//member activated
TID: [0] [STRATOS] [2015-06-04 19:56:00,907]  INFO {org.apache.stratos.cloud.controller.messaging.publisher.TopologyEventPublisher}
-  Publishing member activated event: [service-name] c1 [cluster-id] g-sc-G12-1.c1-0x0.c1.domain
[cluster-instance-id] g-sc-G12-1-1 [member-id] g-sc-G12-1.c1-0x0.c1.domainb0aa0188-49f1-47f6-a040-c2eab4acb5b1
[network-partition-id] RegionOne [partition-id] whole-region
TID: [0] [STRATOS] [2015-06-04 19:56:00,916]  INFO {org.apache.stratos.messaging.message.processor.topology.MemberActivatedMessageProcessor}
-  Member activated: [service] c1 [cluster] g-sc-G12-1.c1-0x0.c1.domain [member] g-sc-G12-1.c1-0x0.c1.domainb0aa0188-49f1-47f6-a040-c2eab4acb5b1

//after 15 minutes ---member is still in pending state, pending timeout expired
TID: [0] [STRATOS] [2015-06-04 20:08:04,713]  INFO {org.apache.stratos.autoscaler.context.partition.ClusterLevelPartitionContext$PendingMemberWatcher}
-  Pending state of member expired, member will be moved to obsolete list. [pending member]
g-sc-G12-1.c1-0x0.c1.domainb0aa0188-49f1-47f6-a040-c2eab4acb5b1 [expiry time] 900000 [cluster]
g-sc-G12-1.c1-0x0.c1.domain [cluster instance] null

On Fri, Jun 5, 2015 at 5:14 AM, Martin Eppel (meppel) <meppel@cisco.com<mailto:meppel@cisco.com>>
wrote:
Hi,

I am running into a scenario where application un-deployment fails (using stratos with latest
commit  b1b6bca3f99b6127da24c9af0a6b20faff2907be).

For application structure see [1.], (debug enabled) wso2carbon.log, application.json, cartridge-group.json,
deployment-policy, auto-scaling policies see attached zip file.

It is noteworthy, that while the application is running the following log statements /exceptions
are observed:

…
Member is in the wrong list and it is removed from active members list: g-sc-G12-1.c1-0x0.c1.domainb0aa0188-49f1-47f6-a040-c2eab4acb5b1
…
TID: [0] [STRATOS] [2015-06-04 20:11:03,425] ERROR {org.apache.stratos.autoscaler.rule.RuleTasksDelegator}
-  Cannot terminate instance
…
// after receiving the application undeploy event:
[2015-06-04 20:12:39,465]  INFO {org.apache.stratos.autoscaler.services.impl.AutoscalerServiceImpl}
-  Application undeployment process started: [application-id] g-sc-G12-1
// a new instance is being started up
…
[2015-06-04 20:13:13,445]  INFO {org.apache.stratos.cloud.controller.services.impl.InstanceCreator}
-  Instance started successfully: [cartridge-type] c2 [cluster-id] g-sc-G12-1.c2-1x0.c2.domain
[instance-id] RegionOne/5d4699f7-b00b-42eb-b565-b48fc8f20407

// Also noteworthy seems the following warning which is seen repeatedly in the logs:
ReadWriteLock} -  System warning! Trying to release a lock which has not been taken by the
same thread: [lock-name]


[1.] Application structure

[cid:image001.png@01D09F76.E90CEC80]







--

Udara Liyanage
Software Engineer
WSO2, Inc.: http://wso2.com<http://wso2.com/>
lean. enterprise. middleware
web: http://udaraliyanage.wordpress.com
phone: +94 71 443 6897<tel:%2B94%2071%20443%206897>



--

Udara Liyanage
Software Engineer
WSO2, Inc.: http://wso2.com<http://wso2.com/>
lean. enterprise. middleware
web: http://udaraliyanage.wordpress.com
phone: +94 71 443 6897<tel:%2B94%2071%20443%206897>



--

Udara Liyanage
Software Engineer
WSO2, Inc.: http://wso2.com<http://wso2.com/>
lean. enterprise. middleware
web: http://udaraliyanage.wordpress.com
phone: +94 71 443 6897



--

Udara Liyanage
Software Engineer
WSO2, Inc.: http://wso2.com<http://wso2.com/>
lean. enterprise. middleware
web: http://udaraliyanage.wordpress.com
phone: +94 71 443 6897
Mime
View raw message