stratos-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lahiru Sandaruwan <lahi...@wso2.com>
Subject Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going active ... was RE: Stratos 4.1.0 : Issue when deploying application with single group and 3 cartridges, no dependencies ...
Date Thu, 02 Apr 2015 08:45:06 GMT
Hi Martin,

Could you please try the fix i have committed?

Thanks,

On Thu, Apr 2, 2015 at 10:35 AM, Martin Eppel (meppel) <meppel@cisco.com>
wrote:

>  Great,
>
>
>
> Thanks
>
>
>
> Martin
>
>
>
> *From:* isuruh@wso2.com [mailto:isuruh@wso2.com] *On Behalf Of *Isuru
> Haththotuwa
> *Sent:* Wednesday, April 01, 2015 9:08 PM
>
> *To:* dev
> *Subject:* Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not
> going active ... was RE: Stratos 4.1.0 : Issue when deploying application
> with single group and 3 cartridges, no dependencies ...
>
>
>
> Hi Lahiru,
>
>
> On Thu, Apr 2, 2015 at 9:11 AM, Lahiru Sandaruwan <lahirus@wso2.com>
> wrote:
>
> Thanks Martin for the logs. So the reason is clear why happens this. We
> keep the payload par cartridge type in iaas provider object, in cloud
> controller context.
>
>
>
> When second member approaches the instance started method, it rewrites the
> payload in iaasProvider object. But the first instance is still not really
> created in iaas, and it runs in a different thread. Therefore the payload
> belong to second instance, is passed to second instance as well as the
> first instance.
>
> Nice work finding this! So the resolution would be to change the cartridge
> type -> payload mapping to alias -> payload?
>
>
>
> I will fix it.
>
>
>
> Thanks.
>
>
>
> On Wed, Apr 1, 2015 at 11:38 PM, Martin Eppel (meppel) <meppel@cisco.com>
> wrote:
>
> Hi Lahiru,
>
>
>
> Attached are the logs,
>
>
>
> Thanks
>
>
>
> Martin
>
>
>
> *From:* Lahiru Sandaruwan [mailto:lahirus@wso2.com]
> *Sent:* Wednesday, April 01, 2015 4:11 AM
>
>
> *To:* dev
> *Subject:* Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not
> going active ... was RE: Stratos 4.1.0 : Issue when deploying application
> with single group and 3 cartridges, no dependencies ...
>
>
>
> HI Martin,
>
>
>
> Could you please send the debug logs enabling debug on following classes?
> I have improved few debug logs to track payload, therefore please take a
> pull before reproducing with debug logs.
>
>
> log4j.logger.org.apache.stratos.autoscaler.applications.parser.DefaultApplicationParser=DEBUG
>
>
> log4j.logger.org.apache.stratos.cloud.controller.services.impl.InstanceCreator=DEBUG
>
>
> log4j.logger.org.apache.stratos.cloud.controller.services.impl.CloudControllerServiceImpl=DEBUG
>
> Thanks.
>
>
>
>
>
> On Wed, Apr 1, 2015 at 10:50 AM, Lahiru Sandaruwan <lahirus@wso2.com>
> wrote:
>
> I will have a look Martin.
>
>
>
> Thanks.
>
>
>
> On Wed, Apr 1, 2015 at 6:54 AM, Martin Eppel (meppel) <meppel@cisco.com>
> wrote:
>
> Hi Lahiru,
>
>
>
> I retested the scenario to make sure proper unique aliases are provided
> and they are, please double check the provided artifacts. Issue remains the
> same, one VM gets stuck in INITIALIZED until terminated and restarted. As
> pointed out before, the cartridge alias in the cartridge agent is the same
> in 2 cartridges (although provided aliases in the application json is
> unique). After restarting the faulty member cartridge agent alias has the
> proper alias and goes into active.
>
> I also provided the cartridge-agent logs for the member which is faulty,
> before restart and after restart (cartridge-agent-70~INITIALIZED, ACTIVE)
>
>
>
> Please see attached logs, artifacts,
>
>
>
> Thanks
>
>
>
> Martin
>
>
>
> *From:* Martin Eppel (meppel)
> *Sent:* Tuesday, March 31, 2015 1:22 PM
>
>
> *To:* dev@stratos.apache.org
> *Subject:* RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not
> going active ... was RE: Stratos 4.1.0 : Issue when deploying application
> with single group and 3 cartridges, no dependencies ...
>
>
>
> Lahiru,
>
>
>
> The aliases are all unique, please check the the application json, and as
> I said in my analysis it I think this is what I saw but I can’t be sure. So
> this is not an issue of starting the application with duplicate aliases but
> rather something else ,
>
>
>
> Thanks
>
>
>
> Martin
>
>
>
> *From:* Lahiru Sandaruwan [mailto:lahirus@wso2.com <lahirus@wso2.com>]
> *Sent:* Tuesday, March 31, 2015 12:58 PM
> *To:* dev
> *Subject:* Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not
> going active ... was RE: Stratos 4.1.0 : Issue when deploying application
> with single group and 3 cartridges, no dependencies ...
>
>
>
> Hi Martin,
>
>
>
> On Wed, Apr 1, 2015 at 12:34 AM, Martin Eppel (meppel) <meppel@cisco.com>
> wrote:
>
> Hi Lahiru,
>
>
>
> I tested the fix and noticed the following issue:
>
>
>
> From the 4 cartridges 3 instances go Active, however one instance got
> stuck in INITIALIZED state.
>
>
>
> One of the things I noticed while the member was stuck in INITIALIZED
> state is that the system received 2 active events for the same membered:
>
>
>
> *TID: [0] [STRATOS] [2015-03-31 18:10:27,645]  INFO
> {org.apache.stratos.cloud.controller.messaging.publisher.TopologyEventPublisher}
> -  Publishing member activated event: [service-name] c2 [cluster-id]
> subscription-G1-G2-G3.c2-1x1.c2.domain [cluster-instance-id]
> subscription-G1-G2-G3-1 [member-id]
> subscription-G1-G2-G3.c2-1x1.c2.domainf939e8dc-e0cc-408a-8343-ac86e479549f
> [network-partition-id] RegionOne [partition-id] whole-region*
>
> *TID: [0] [STRATOS] [2015-03-31 18:10:31,059]  INFO
> {org.apache.stratos.cloud.controller.messaging.publisher.TopologyEventPublisher}
> -  Publishing member activated event: [service-name] c2 [cluster-id]
> subscription-G1-G2-G3.c2-1x1.c2.domain [cluster-instance-id]
> subscription-G1-G2-G3-1 [member-id]
> subscription-G1-G2-G3.c2-1x1.c2.domainf939e8dc-e0cc-408a-8343-ac86e479549f
> [network-partition-id] RegionOne [partition-id] whole-region*
>
>
>
> As another note, but I can’t be sure since the log got overwritten is that
> it appeared to me that at one point (while the cartridge was stuck in
> INITIALIZED state is that the cartridge alias of 2 cartridges was the same
> (~71, ~70: 71: CARTRIDGE_ALIAS=c2-1x1), but I can’t say for sure since when
> I double checked a new instance of the cartridge was created (which went
> ACTIVE).
>
>
>
>
>
> Can i assume this occurred when two cartridges use same alias? May be we
> have to put a validation on Application as we cannot use same alias in two
> cartridges.
>
>
>
> If so, could you please try to reproduce this with different aliases?
>
>
>
> Thanks.
>
>
>
>
>
>
>
> Eventually the system terminated the “faulty” member and the cartridge
> came up and went into active state:
>
>
>
> *“Pending state of member expired, member will be moved to obsolete list.
> [pending member]
> subscription-G1-G2-G3.c2-1x0.c2.domainfe9a30cf-fffc-4a6c-9b3b-f419498895b9*
> ”
>
> …
>
> “*Member activated: [service] c2 [cluster]
> subscription-G1-G2-G3.c2-1x0.c2.domain [member]
> subscription-G1-G2-G3.c2-1x0.c2.domain44a2ae3c-9968-41de-9e10-7212ef3788c2*
> ”
>
>
>
> Cartridge alias after all 4 cartridges went ACTIVE:
>
>
>
> 70: CARTRIDGE_ALIAS=c2-1x0
>
> 71: CARTRIDGE_ALIAS=c2-1x1
>
>
>
> 69: CARTRIDGE_ALIAS=c1-1x0
>
> 68: CARTRIDGE_ALIAS=c1-1x1
>
>
>
> I attached the logs and artefacts
>
>
>
> Thanks
>
>
>
> Martin
>
>
>
> *From:* Lahiru Sandaruwan [mailto:lahirus@wso2.com]
> *Sent:* Tuesday, March 31, 2015 2:24 AM
>
>
> *To:* dev
> *Subject:* Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not
> going active ... was RE: Stratos 4.1.0 : Issue when deploying application
> with single group and 3 cartridges, no dependencies ...
>
>
>
> Hi Martin,
>
>
>
> I have reproduced the issue and pushed a fix. Please verify.
>
>
>
> Thanks.
>
>
>
> On Tue, Mar 31, 2015 at 8:32 AM, Lahiru Sandaruwan <lahirus@wso2.com>
> wrote:
>
> Hi Martin,
>
>
>
> On Tue, Mar 31, 2015 at 5:41 AM, Martin Eppel (meppel) <meppel@cisco.com>
> wrote:
>
> Hi Lahiru,
>
>
>
> I tested a new scenario, where I have 2 groups with cartridges and no
> start up dependencies and I see again the issue with cartridge instances
> being stuck in “started” state. I also noticed the following log message:
>
> *“Artifact repositories not found for application signup, artifact updated
> event not sent: [application-id] subscription-G1-G2-G3*”.
>
>
>
>
>
> Will have a look.
>
>
>
> Thanks.
>
>  Please find attached the artefacts and log messages,
>
>
>
> Thanks
>
>
>
> Martin
>
>
>
> *From:* Martin Eppel (meppel)
> *Sent:* Monday, March 30, 2015 2:36 PM
>
>
> *To:* dev@stratos.apache.org
> *Subject:* RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not
> going active ... was RE: Stratos 4.1.0 : Issue when deploying application
> with single group and 3 cartridges, no dependencies ...
>
>
>
> Hi Lahiru,
>
>
>
> I tested both scenarios,
>
>
>
> 1.      Single cartridge in single group and it worked, cartridge
> instance starts up and goes active,
>
> 2.      Multiple cartridges with startup dependencies :
>
> All instances are started up (in right sequence) and go active
>
> JIRA https://issues.apache.org/jira/browse/STRATOS-1292[image:
> http://wikicentral.cisco.com/images/icons/linkext7.gif]
> <https://issues.apache.org/jira/browse/STRATOS-1292> is being tested and
> verified
>
>
>
> Removing / Un-deploying the application caused an issue as one of the VMs
> did not get terminated (throwing exceptions in the log). I am not sure if
> this is already a known issue  related to to the ongoing discussion of
> application removal / un-deployment.
>
> Just in case, I zipped up the logs and attached it to the email thread
> (single-group-multiple-cartridges-startup.zip). Let me know if the
> un-deployment issue is the same already discussed or if this is a new issue
> ?
>
>
>
> Same error occurred in the scenario where multiple cartridges were started
> up (all go active) and then application was removed, I attached the log for
> this scenario as well
> (wscarbon-application-single-group-multiple-cartridges-no-startup-remval-error.log.zip)
>
>
>
>
>
> Thanks
>
>
>
> Martin
>
>
>
>
>
> *From:* Martin Eppel (meppel)
> *Sent:* Monday, March 30, 2015 7:47 AM
> *To:* dev@stratos.apache.org
> *Subject:* RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not
> going active ... was RE: Stratos 4.1.0 : Issue when deploying application
> with single group and 3 cartridges, no dependencies ...
>
>
>
> Thanks Lahiru,
>
>
>
> Will do and let you know
>
>
>
> Martin
>
>
>
> *From:* Lahiru Sandaruwan [mailto:lahirus@wso2.com <lahirus@wso2.com>]
> *Sent:* Monday, March 30, 2015 3:32 AM
> *To:* dev
> *Subject:* Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not
> going active ... was RE: Stratos 4.1.0 : Issue when deploying application
> with single group and 3 cartridges, no dependencies ...
>
>
>
> Hi Martin,
>
>
>
> I was able to reproduce the issue and committed a fix.
>
> Could you please verify?
>
>
>
> Thanks.
>
>
>
> On Sun, Mar 29, 2015 at 5:43 PM, Lahiru Sandaruwan <lahirus@wso2.com>
> wrote:
>
> Hi Martin,
>
>
>
> According to the logs attached, it seems the cartridge agent waits for the
> artifact updated event, and somehow it is not sent.
>
>
>
> Looking into it.
>
>
>
> Thanks.
>
>
>
> On Sat, Mar 28, 2015 at 6:48 AM, Imesh Gunaratne <imesh@apache.org> wrote:
>
> Hi Martin,
>
>
>
> Thanks for the quick response on this!
>
> I will use these artifacts to re-produce the problem locally and will let
> you know the results soon.
>
>
>
> Thanks
>
>
>
> On Sat, Mar 28, 2015 at 4:19 AM, Martin Eppel (meppel) <meppel@cisco.com>
> wrote:
>
> Hi Imesh,
>
>
>
> As discussed please find attached the “real” artefacts (application,
> cartridge group, cartridge):
>
>
>
> Attached it as well to the jira
>
>
>
> Thanks
>
>
>
> Martin
>
>
>
>
>
> *From:* Martin Eppel (meppel)
> *Sent:* Friday, March 27, 2015 11:37 AM
> *To:* dev@stratos.apache.org
> *Subject:* RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not
> going active ... was RE: Stratos 4.1.0 : Issue when deploying application
> with single group and 3 cartridges, no dependencies ...
>
>
>
> Hi Imesh,
>
>
>
> Originally I tested with 3 cartridges but then realized that it also
> occurs with a single cartridge in a group – see also the JIRA for the
> respective notes and artefacts (single group, single cartridge)
>
>
>
> Thanks
>
>
>
> Martin
>
>
>
> *From:* Imesh Gunaratne [mailto:imesh@apache.org <imesh@apache.org>]
> *Sent:* Friday, March 27, 2015 11:34 AM
> *To:* dev
> *Subject:* Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not
> going active ... was RE: Stratos 4.1.0 : Issue when deploying application
> with single group and 3 cartridges, no dependencies ...
>
>
>
> Hi Martin,
>
>
>
> Thanks for reporting this problem. You have mentioned that there are three
> cartridges in the application but in the given application.json I only can
> see one cartridge. Are we missing anything here? Will it be possible for
> you to attach the complete set of Stratos artifacts that you are using in a
> zip file?
>
>
>
> Thanks
>
> Imesh
>
>
>
> On Fri, Mar 27, 2015 at 11:51 PM, Martin Eppel (meppel) <meppel@cisco.com>
> wrote:
>
> Forgot to mention, this is the same issue as reported in JIRA
> https://issues.apache.org/jira/browse/STRATOS-1292
>
>
>
> Thanks
>
>
>
> Martin
>
>
>
> *From:* Martin Eppel (meppel)
> *Sent:* Friday, March 27, 2015 10:52 AM
> *To:* dev@stratos.apache.org
> *Subject:* Issue (?): Stratos 4.1.0 - Cartridges in a group are not going
> active ... was RE: Stratos 4.1.0 : Issue when deploying application with
> single group and 3 cartridges, no dependencies ...
>
>
>
> I retested with a similar scenario (single cartridges in group)  with
> debug logs turned on, please see logs attached.
>
>
>
> Cartridge is stuck in “Starting” state
>
>
>
> Thanks
>
>
>
> Martin
>
>
>
> *From:* Martin Eppel (meppel)
> *Sent:* Monday, March 23, 2015 7:15 PM
> *To:* dev@stratos.apache.org
> *Subject:* Stratos 4.1.0 : Issue when deploying application with single
> group and 3 cartridges, no dependencies ...
>
>
>
> Hi,
>
>
>
> I see an issue when I subscribe a single group with 3 cartridges, no
> dependencies – they never go active instead they all are stuck in
>  “Starting”. I attached the server log and the cartridge-agent logs
>
>
>
>
>
> Member status:
>
> subscription-c1-c2-c3: applicationInstances 1, groupInstances 1,
> clusterInstances 3, members 3 (Starting 3)
>
>                   c2: 172.16.2.18/172.28.40.69: status Starting
>
>                   c3: 172.16.2.19/172.28.40.70: status Starting
>
>                   c1: 172.16.2.17/172.28.40.68: status Starting
>
>
>
> Let me know  if I should go ahead and open a JIRA
>
>
>
>
>
> Thanks
>
>
>
> Martin
>
>
>
>
>
> --
>
> Imesh Gunaratne
>
>
>
> Technical Lead, WSO2
>
> Committer & PMC Member, Apache Stratos
>
>
>
>
>
> --
>
> Imesh Gunaratne
>
>
>
> Technical Lead, WSO2
>
> Committer & PMC Member, Apache Stratos
>
>
>
>
>
> --
>
> --
> Lahiru Sandaruwan
>
> Committer and PMC member, Apache Stratos,
> Senior Software Engineer,
> WSO2 Inc., http://wso2.com
>
> lean.enterprise.middleware
>
> phone: +94773325954
> email: lahirus@wso2.com blog: http://lahiruwrites.blogspot.com/
> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>
>
>
>
>
>
>
> --
>
> --
> Lahiru Sandaruwan
>
> Committer and PMC member, Apache Stratos,
> Senior Software Engineer,
> WSO2 Inc., http://wso2.com
>
> lean.enterprise.middleware
>
> phone: +94773325954
> email: lahirus@wso2.com blog: http://lahiruwrites.blogspot.com/
> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>
>
>
>
>
>
>
> --
>
> --
> Lahiru Sandaruwan
>
> Committer and PMC member, Apache Stratos,
> Senior Software Engineer,
> WSO2 Inc., http://wso2.com
>
> lean.enterprise.middleware
>
> phone: +94773325954
> email: lahirus@wso2.com blog: http://lahiruwrites.blogspot.com/
> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>
>
>
>
>
>
>
> --
>
> --
> Lahiru Sandaruwan
>
> Committer and PMC member, Apache Stratos,
> Senior Software Engineer,
> WSO2 Inc., http://wso2.com
>
> lean.enterprise.middleware
>
> phone: +94773325954
> email: lahirus@wso2.com blog: http://lahiruwrites.blogspot.com/
> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>
>
>
>
>
>
>
> --
>
> --
> Lahiru Sandaruwan
>
> Committer and PMC member, Apache Stratos,
> Senior Software Engineer,
> WSO2 Inc., http://wso2.com
>
> lean.enterprise.middleware
>
> phone: +94773325954
> email: lahirus@wso2.com blog: http://lahiruwrites.blogspot.com/
> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>
>
>
>
>
>
>
> --
>
> --
> Lahiru Sandaruwan
>
> Committer and PMC member, Apache Stratos,
> Senior Software Engineer,
> WSO2 Inc., http://wso2.com
>
> lean.enterprise.middleware
>
> phone: +94773325954
> email: lahirus@wso2.com blog: http://lahiruwrites.blogspot.com/
> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>
>
>
>
>
>
>
> --
>
> --
> Lahiru Sandaruwan
>
> Committer and PMC member, Apache Stratos,
> Senior Software Engineer,
> WSO2 Inc., http://wso2.com
>
> lean.enterprise.middleware
>
> phone: +94773325954
> email: lahirus@wso2.com blog: http://lahiruwrites.blogspot.com/
> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>
>
>
>
>
>
>
> --
>
> --
> Lahiru Sandaruwan
>
> Committer and PMC member, Apache Stratos,
> Senior Software Engineer,
> WSO2 Inc., http://wso2.com
>
> lean.enterprise.middleware
>
> phone: +94773325954
> email: lahirus@wso2.com blog: http://lahiruwrites.blogspot.com/
> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>
> --
>  <http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146>
>
>  <http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146>
>
>
>
>
>
> *Thanks and Regards, Isuru H.
> <http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146>*
>
>
>
> +94 716 358 048 <http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146>
>
>
>
>


-- 
--
Lahiru Sandaruwan
Committer and PMC member, Apache Stratos,
Senior Software Engineer,
WSO2 Inc., http://wso2.com
lean.enterprise.middleware

phone: +94773325954
email: lahirus@wso2.com blog: http://lahiruwrites.blogspot.com/
linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146

Mime
View raw message