stratos-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Martin Eppel (meppel)" <mep...@cisco.com>
Subject RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going active ... was RE: Stratos 4.1.0 : Issue when deploying application with single group and 3 cartridges, no dependencies ...
Date Thu, 02 Apr 2015 20:24:36 GMT
Hi Lahiru,

Tested the fix and it works, nice work !

I attached he log files in case you want to double check,

Regards

Martin

From: Lahiru Sandaruwan [mailto:lahirus@wso2.com]
Sent: Thursday, April 02, 2015 1:45 AM
To: dev
Subject: Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going active ... was
RE: Stratos 4.1.0 : Issue when deploying application with single group and 3 cartridges, no
dependencies ...

Hi Martin,

Could you please try the fix i have committed?

Thanks,

On Thu, Apr 2, 2015 at 10:35 AM, Martin Eppel (meppel) <meppel@cisco.com<mailto:meppel@cisco.com>>
wrote:
Great,

Thanks

Martin

From: isuruh@wso2.com<mailto:isuruh@wso2.com> [mailto:isuruh@wso2.com<mailto:isuruh@wso2.com>]
On Behalf Of Isuru Haththotuwa
Sent: Wednesday, April 01, 2015 9:08 PM

To: dev
Subject: Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going active ... was
RE: Stratos 4.1.0 : Issue when deploying application with single group and 3 cartridges, no
dependencies ...

Hi Lahiru,

On Thu, Apr 2, 2015 at 9:11 AM, Lahiru Sandaruwan <lahirus@wso2.com<mailto:lahirus@wso2.com>>
wrote:
Thanks Martin for the logs. So the reason is clear why happens this. We keep the payload par
cartridge type in iaas provider object, in cloud controller context.

When second member approaches the instance started method, it rewrites the payload in iaasProvider
object. But the first instance is still not really created in iaas, and it runs in a different
thread. Therefore the payload belong to second instance, is passed to second instance as well
as the first instance.
Nice work finding this! So the resolution would be to change the cartridge type -> payload
mapping to alias -> payload?

I will fix it.

Thanks.

On Wed, Apr 1, 2015 at 11:38 PM, Martin Eppel (meppel) <meppel@cisco.com<mailto:meppel@cisco.com>>
wrote:
Hi Lahiru,

Attached are the logs,

Thanks

Martin

From: Lahiru Sandaruwan [mailto:lahirus@wso2.com<mailto:lahirus@wso2.com>]
Sent: Wednesday, April 01, 2015 4:11 AM

To: dev
Subject: Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going active ... was
RE: Stratos 4.1.0 : Issue when deploying application with single group and 3 cartridges, no
dependencies ...

HI Martin,

Could you please send the debug logs enabling debug on following classes? I have improved
few debug logs to track payload, therefore please take a pull before reproducing with debug
logs.
log4j.logger.org.apache.stratos.autoscaler.applications.parser.DefaultApplicationParser=DEBUG
log4j.logger.org.apache.stratos.cloud.controller.services.impl.InstanceCreator=DEBUG
log4j.logger.org.apache.stratos.cloud.controller.services.impl.CloudControllerServiceImpl=DEBUG
Thanks.


On Wed, Apr 1, 2015 at 10:50 AM, Lahiru Sandaruwan <lahirus@wso2.com<mailto:lahirus@wso2.com>>
wrote:
I will have a look Martin.

Thanks.

On Wed, Apr 1, 2015 at 6:54 AM, Martin Eppel (meppel) <meppel@cisco.com<mailto:meppel@cisco.com>>
wrote:
Hi Lahiru,

I retested the scenario to make sure proper unique aliases are provided and they are, please
double check the provided artifacts. Issue remains the same, one VM gets stuck in INITIALIZED
until terminated and restarted. As pointed out before, the cartridge alias in the cartridge
agent is the same in 2 cartridges (although provided aliases in the application json is unique).
After restarting the faulty member cartridge agent alias has the proper alias and goes into
active.
I also provided the cartridge-agent logs for the member which is faulty, before restart and
after restart (cartridge-agent-70~INITIALIZED, ACTIVE)

Please see attached logs, artifacts,

Thanks

Martin

From: Martin Eppel (meppel)
Sent: Tuesday, March 31, 2015 1:22 PM

To: dev@stratos.apache.org<mailto:dev@stratos.apache.org>
Subject: RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going active ... was
RE: Stratos 4.1.0 : Issue when deploying application with single group and 3 cartridges, no
dependencies ...

Lahiru,

The aliases are all unique, please check the the application json, and as I said in my analysis
it I think this is what I saw but I can’t be sure. So this is not an issue of starting the
application with duplicate aliases but rather something else ,

Thanks

Martin

From: Lahiru Sandaruwan [mailto:lahirus@wso2.com]
Sent: Tuesday, March 31, 2015 12:58 PM
To: dev
Subject: Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going active ... was
RE: Stratos 4.1.0 : Issue when deploying application with single group and 3 cartridges, no
dependencies ...

Hi Martin,

On Wed, Apr 1, 2015 at 12:34 AM, Martin Eppel (meppel) <meppel@cisco.com<mailto:meppel@cisco.com>>
wrote:
Hi Lahiru,

I tested the fix and noticed the following issue:

From the 4 cartridges 3 instances go Active, however one instance got stuck in INITIALIZED
state.

One of the things I noticed while the member was stuck in INITIALIZED state is that the system
received 2 active events for the same membered:

TID: [0] [STRATOS] [2015-03-31 18:10:27,645]  INFO {org.apache.stratos.cloud.controller.messaging.publisher.TopologyEventPublisher}
-  Publishing member activated event: [service-name] c2 [cluster-id] subscription-G1-G2-G3.c2-1x1.c2.domain
[cluster-instance-id] subscription-G1-G2-G3-1 [member-id] subscription-G1-G2-G3.c2-1x1.c2.domainf939e8dc-e0cc-408a-8343-ac86e479549f
[network-partition-id] RegionOne [partition-id] whole-region
TID: [0] [STRATOS] [2015-03-31 18:10:31,059]  INFO {org.apache.stratos.cloud.controller.messaging.publisher.TopologyEventPublisher}
-  Publishing member activated event: [service-name] c2 [cluster-id] subscription-G1-G2-G3.c2-1x1.c2.domain
[cluster-instance-id] subscription-G1-G2-G3-1 [member-id] subscription-G1-G2-G3.c2-1x1.c2.domainf939e8dc-e0cc-408a-8343-ac86e479549f
[network-partition-id] RegionOne [partition-id] whole-region

As another note, but I can’t be sure since the log got overwritten is that it appeared to
me that at one point (while the cartridge was stuck in INITIALIZED state is that the cartridge
alias of 2 cartridges was the same (~71, ~70: 71: CARTRIDGE_ALIAS=c2-1x1), but I can’t say
for sure since when I double checked a new instance of the cartridge was created (which went
ACTIVE).


Can i assume this occurred when two cartridges use same alias? May be we have to put a validation
on Application as we cannot use same alias in two cartridges.

If so, could you please try to reproduce this with different aliases?

Thanks.



Eventually the system terminated the “faulty” member and the cartridge came up and went
into active state:

“Pending state of member expired, member will be moved to obsolete list. [pending member]
subscription-G1-G2-G3.c2-1x0.c2.domainfe9a30cf-fffc-4a6c-9b3b-f419498895b9”
…
“Member activated: [service] c2 [cluster] subscription-G1-G2-G3.c2-1x0.c2.domain [member]
subscription-G1-G2-G3.c2-1x0.c2.domain44a2ae3c-9968-41de-9e10-7212ef3788c2”

Cartridge alias after all 4 cartridges went ACTIVE:

70: CARTRIDGE_ALIAS=c2-1x0
71: CARTRIDGE_ALIAS=c2-1x1

69: CARTRIDGE_ALIAS=c1-1x0
68: CARTRIDGE_ALIAS=c1-1x1

I attached the logs and artefacts

Thanks

Martin

From: Lahiru Sandaruwan [mailto:lahirus@wso2.com<mailto:lahirus@wso2.com>]
Sent: Tuesday, March 31, 2015 2:24 AM

To: dev
Subject: Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going active ... was
RE: Stratos 4.1.0 : Issue when deploying application with single group and 3 cartridges, no
dependencies ...

Hi Martin,

I have reproduced the issue and pushed a fix. Please verify.

Thanks.

On Tue, Mar 31, 2015 at 8:32 AM, Lahiru Sandaruwan <lahirus@wso2.com<mailto:lahirus@wso2.com>>
wrote:
Hi Martin,

On Tue, Mar 31, 2015 at 5:41 AM, Martin Eppel (meppel) <meppel@cisco.com<mailto:meppel@cisco.com>>
wrote:
Hi Lahiru,

I tested a new scenario, where I have 2 groups with cartridges and no start up dependencies
and I see again the issue with cartridge instances being stuck in “started” state. I also
noticed the following log message:
“Artifact repositories not found for application signup, artifact updated event not sent:
[application-id] subscription-G1-G2-G3”.


Will have a look.

Thanks.
Please find attached the artefacts and log messages,

Thanks

Martin

From: Martin Eppel (meppel)
Sent: Monday, March 30, 2015 2:36 PM

To: dev@stratos.apache.org<mailto:dev@stratos.apache.org>
Subject: RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going active ... was
RE: Stratos 4.1.0 : Issue when deploying application with single group and 3 cartridges, no
dependencies ...

Hi Lahiru,

I tested both scenarios,


1.      Single cartridge in single group and it worked, cartridge instance starts up and goes
active,

2.      Multiple cartridges with startup dependencies :

All instances are started up (in right sequence) and go active

JIRA https://issues.apache.org/jira/browse/STRATOS-1292[http://wikicentral.cisco.com/images/icons/linkext7.gif]
is being tested and verified

Removing / Un-deploying the application caused an issue as one of the VMs did not get terminated
(throwing exceptions in the log). I am not sure if this is already a known issue  related
to to the ongoing discussion of application removal / un-deployment.
Just in case, I zipped up the logs and attached it to the email thread (single-group-multiple-cartridges-startup.zip).
Let me know if the un-deployment issue is the same already discussed or if this is a new issue
?

Same error occurred in the scenario where multiple cartridges were started up (all go active)
and then application was removed, I attached the log for this scenario as well (wscarbon-application-single-group-multiple-cartridges-no-startup-remval-error.log.zip)


Thanks

Martin


From: Martin Eppel (meppel)
Sent: Monday, March 30, 2015 7:47 AM
To: dev@stratos.apache.org<mailto:dev@stratos.apache.org>
Subject: RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going active ... was
RE: Stratos 4.1.0 : Issue when deploying application with single group and 3 cartridges, no
dependencies ...

Thanks Lahiru,

Will do and let you know

Martin

From: Lahiru Sandaruwan [mailto:lahirus@wso2.com]
Sent: Monday, March 30, 2015 3:32 AM
To: dev
Subject: Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going active ... was
RE: Stratos 4.1.0 : Issue when deploying application with single group and 3 cartridges, no
dependencies ...

Hi Martin,

I was able to reproduce the issue and committed a fix.
Could you please verify?

Thanks.

On Sun, Mar 29, 2015 at 5:43 PM, Lahiru Sandaruwan <lahirus@wso2.com<mailto:lahirus@wso2.com>>
wrote:
Hi Martin,

According to the logs attached, it seems the cartridge agent waits for the artifact updated
event, and somehow it is not sent.

Looking into it.

Thanks.

On Sat, Mar 28, 2015 at 6:48 AM, Imesh Gunaratne <imesh@apache.org<mailto:imesh@apache.org>>
wrote:
Hi Martin,

Thanks for the quick response on this!
I will use these artifacts to re-produce the problem locally and will let you know the results
soon.

Thanks

On Sat, Mar 28, 2015 at 4:19 AM, Martin Eppel (meppel) <meppel@cisco.com<mailto:meppel@cisco.com>>
wrote:
Hi Imesh,

As discussed please find attached the “real” artefacts (application, cartridge group,
cartridge):

Attached it as well to the jira

Thanks

Martin


From: Martin Eppel (meppel)
Sent: Friday, March 27, 2015 11:37 AM
To: dev@stratos.apache.org<mailto:dev@stratos.apache.org>
Subject: RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going active ... was
RE: Stratos 4.1.0 : Issue when deploying application with single group and 3 cartridges, no
dependencies ...

Hi Imesh,

Originally I tested with 3 cartridges but then realized that it also occurs with a single
cartridge in a group – see also the JIRA for the respective notes and artefacts (single
group, single cartridge)

Thanks

Martin

From: Imesh Gunaratne [mailto:imesh@apache.org]
Sent: Friday, March 27, 2015 11:34 AM
To: dev
Subject: Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going active ... was
RE: Stratos 4.1.0 : Issue when deploying application with single group and 3 cartridges, no
dependencies ...

Hi Martin,

Thanks for reporting this problem. You have mentioned that there are three cartridges in the
application but in the given application.json I only can see one cartridge. Are we missing
anything here? Will it be possible for you to attach the complete set of Stratos artifacts
that you are using in a zip file?

Thanks
Imesh

On Fri, Mar 27, 2015 at 11:51 PM, Martin Eppel (meppel) <meppel@cisco.com<mailto:meppel@cisco.com>>
wrote:
Forgot to mention, this is the same issue as reported in JIRA https://issues.apache.org/jira/browse/STRATOS-1292

Thanks

Martin

From: Martin Eppel (meppel)
Sent: Friday, March 27, 2015 10:52 AM
To: dev@stratos.apache.org<mailto:dev@stratos.apache.org>
Subject: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going active ... was RE:
Stratos 4.1.0 : Issue when deploying application with single group and 3 cartridges, no dependencies
...

I retested with a similar scenario (single cartridges in group)  with debug logs turned on,
please see logs attached.

Cartridge is stuck in “Starting” state

Thanks

Martin

From: Martin Eppel (meppel)
Sent: Monday, March 23, 2015 7:15 PM
To: dev@stratos.apache.org<mailto:dev@stratos.apache.org>
Subject: Stratos 4.1.0 : Issue when deploying application with single group and 3 cartridges,
no dependencies ...

Hi,

I see an issue when I subscribe a single group with 3 cartridges, no dependencies – they
never go active instead they all are stuck in  “Starting”. I attached the server log and
the cartridge-agent logs


Member status:
subscription-c1-c2-c3: applicationInstances 1, groupInstances 1, clusterInstances 3, members
3 (Starting 3)
                  c2: 172.16.2.18/172.28.40.69<http://172.16.2.18/172.28.40.69>: status
Starting
                  c3: 172.16.2.19/172.28.40.70<http://172.16.2.19/172.28.40.70>: status
Starting
                  c1: 172.16.2.17/172.28.40.68<http://172.16.2.17/172.28.40.68>: status
Starting

Let me know  if I should go ahead and open a JIRA


Thanks

Martin



--
Imesh Gunaratne

Technical Lead, WSO2
Committer & PMC Member, Apache Stratos



--
Imesh Gunaratne

Technical Lead, WSO2
Committer & PMC Member, Apache Stratos



--
--
Lahiru Sandaruwan
Committer and PMC member, Apache Stratos,
Senior Software Engineer,
WSO2 Inc., http://wso2.com
lean.enterprise.middleware

phone: +94773325954<tel:%2B94773325954>
email: lahirus@wso2.com<mailto:lahirus@wso2.com> blog: http://lahiruwrites.blogspot.com/
linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146




--
--
Lahiru Sandaruwan
Committer and PMC member, Apache Stratos,
Senior Software Engineer,
WSO2 Inc., http://wso2.com
lean.enterprise.middleware

phone: +94773325954<tel:%2B94773325954>
email: lahirus@wso2.com<mailto:lahirus@wso2.com> blog: http://lahiruwrites.blogspot.com/
linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146




--
--
Lahiru Sandaruwan
Committer and PMC member, Apache Stratos,
Senior Software Engineer,
WSO2 Inc., http://wso2.com
lean.enterprise.middleware

phone: +94773325954<tel:%2B94773325954>
email: lahirus@wso2.com<mailto:lahirus@wso2.com> blog: http://lahiruwrites.blogspot.com/
linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146




--
--
Lahiru Sandaruwan
Committer and PMC member, Apache Stratos,
Senior Software Engineer,
WSO2 Inc., http://wso2.com
lean.enterprise.middleware

phone: +94773325954<tel:%2B94773325954>
email: lahirus@wso2.com<mailto:lahirus@wso2.com> blog: http://lahiruwrites.blogspot.com/
linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146




--
--
Lahiru Sandaruwan
Committer and PMC member, Apache Stratos,
Senior Software Engineer,
WSO2 Inc., http://wso2.com
lean.enterprise.middleware

phone: +94773325954<tel:%2B94773325954>
email: lahirus@wso2.com<mailto:lahirus@wso2.com> blog: http://lahiruwrites.blogspot.com/
linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146




--
--
Lahiru Sandaruwan
Committer and PMC member, Apache Stratos,
Senior Software Engineer,
WSO2 Inc., http://wso2.com
lean.enterprise.middleware

phone: +94773325954<tel:%2B94773325954>
email: lahirus@wso2.com<mailto:lahirus@wso2.com> blog: http://lahiruwrites.blogspot.com/
linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146




--
--
Lahiru Sandaruwan
Committer and PMC member, Apache Stratos,
Senior Software Engineer,
WSO2 Inc., http://wso2.com
lean.enterprise.middleware

phone: +94773325954<tel:%2B94773325954>
email: lahirus@wso2.com<mailto:lahirus@wso2.com> blog: http://lahiruwrites.blogspot.com/
linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146




--
--
Lahiru Sandaruwan
Committer and PMC member, Apache Stratos,
Senior Software Engineer,
WSO2 Inc., http://wso2.com
lean.enterprise.middleware

phone: +94773325954<tel:%2B94773325954>
email: lahirus@wso2.com<mailto:lahirus@wso2.com> blog: http://lahiruwrites.blogspot.com/
linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146

--
<http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146>
Thanks and Regards,

Isuru H.

<http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146>

+94 716 358 048<http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146>




--
--
Lahiru Sandaruwan
Committer and PMC member, Apache Stratos,
Senior Software Engineer,
WSO2 Inc., http://wso2.com
lean.enterprise.middleware

phone: +94773325954
email: lahirus@wso2.com<mailto:lahirus@wso2.com> blog: http://lahiruwrites.blogspot.com/
linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146

Mime
View raw message