Return-Path: X-Original-To: apmail-stratos-dev-archive@minotaur.apache.org Delivered-To: apmail-stratos-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id D0AE9175C5 for ; Thu, 2 Apr 2015 17:09:16 +0000 (UTC) Received: (qmail 5674 invoked by uid 500); 2 Apr 2015 17:09:16 -0000 Delivered-To: apmail-stratos-dev-archive@stratos.apache.org Received: (qmail 5632 invoked by uid 500); 2 Apr 2015 17:09:16 -0000 Mailing-List: contact dev-help@stratos.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@stratos.apache.org Delivered-To: mailing list dev@stratos.apache.org Received: (qmail 5622 invoked by uid 99); 2 Apr 2015 17:09:16 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Apr 2015 17:09:16 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED,T_RP_MATCHES_RCVD X-Spam-Check-By: apache.org Received: from [140.211.11.3] (HELO mail.apache.org) (140.211.11.3) by apache.org (qpsmtpd/0.29) with SMTP; Thu, 02 Apr 2015 17:09:13 +0000 Received: (qmail 4206 invoked by uid 99); 2 Apr 2015 17:08:53 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Apr 2015 17:08:53 +0000 Date: Thu, 2 Apr 2015 17:08:53 +0000 (UTC) From: "Jeffrey Nguyen (JIRA)" To: dev@stratos.incubator.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (STRATOS-1293) Stratos should remove instance in ERROR state before trying to re-launch instance MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/STRATOS-1293?page=3Dcom.atlassi= an.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D14= 392962#comment-14392962 ]=20 Jeffrey Nguyen commented on STRATOS-1293: ----------------------------------------- Hi Anuruddha, The instances that are in Error state on Openstack Horizon were never remov= ed even after Stratos successfully spawned an instance. It sounds like yo= u might need to enhance jClouds API to return an object with nodeID info fo= r this case. Or perhaps a better solution would be to modify the jClouds A= PI to delete the failed instance if it wasn=C2=B9t spawned successfully (or= make that an option of the API that handles spawning new instance). Regards, -Jeffrey > Stratos should remove instance in ERROR state before trying to re-launch = instance > -------------------------------------------------------------------------= -------- > > Key: STRATOS-1293 > URL: https://issues.apache.org/jira/browse/STRATOS-1293 > Project: Stratos > Issue Type: Bug > Components: Cloud Controller > Affects Versions: 4.0.0, 4.1.0 Alpha > Environment: Openstack Icehouse, Stratos 4.0.0 GA > Reporter: Jeffrey Nguyen > > On my setup with Icehouse and Stratos 4.0.0GA, I observed there was one p= articular cartridge with one running instance and multiple instances in ERR= OR state. Upon checking wso2carbon.log, I found several instances of the = exception below. Looked like when Stratos launched the cartridge, the ins= tance didn't achieve running state, so Stratos tried to launch another inst= ance. This kept on going until eventually one instance of the cartridge a= chieved running status. =20 > We need to make sure when this condition occurs, Stratos will remove the = instances that are in ERROR state before attempting to re-launch. The inst= ances in ERROR state can exhaust resources on the underline Iaas cluster (O= penstack in this case) > 1) IllegalStateException on node RegionOne/887351d5-2c16-48b9-927e-0ca5f1= 3fcc10:java.lang.IllegalStateException: node(RegionOne/887351d5-2c16-48b9-9= 27e-0ca5f13fcc10) didn't achieve the status running; aborting after 1 secon= ds with final status: ERROR > at org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRu= nning.java:72) > at org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRu= nning.java:45) at org.jclouds.compute.strategy.CustomizeNodeAndAddTo= GoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExcep= tionIntoBadMap.java:121) at org.jclouds.compute.strategy.CustomizeNo= deAndAddToGoodMapOrPutExceptionIntoBadMap.apply(CustomizeNodeAndAddToGoodMa= pOrPutExceptionIntoBadMap.java:146) at org.jclouds.compute.strategy.= CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.apply(CustomizeNodeAnd= AddToGoodMapOrPutExceptionIntoBadMap.java:53) > at com.google.common.util.concurrent.Futures$1.apply(Futures.java= :711) at com.google.common.util.concurrent.Futures$ChainingListenabl= eFuture.run(Futures.java:849) at java.util.concurrent.ThreadPoolExec= utor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent= .ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.= lang.Thread.run(Thread.java:745) > 1 error[s] at org.apache.stratos.autoscaler.client.cloud.controlle= r.CloudControllerClient.spawnAnInstance(CloudControllerClient.java:174) > at org.apache.stratos.autoscaler.rule.RuleTasksDelegator.delegate= Spawn(RuleTasksDelegator.java:87) > ... 22 moreCaused by: org.apache.axis2.AxisFault: Failed to start= an instance. MemberContext [memberId=3Dlb01.lb01.domaincde68af9-d82f-42c3-= 9c01-4fd8da565867, nodeId=3Dnull, clusterId=3Dlb01.lb01.domain, cartridgeTy= pe=3Dlb01, privateIpAddresses=3Dnull, publicIpAddresses=3Dnull, allocatedIp= Address=3Dnull, initTime=3D1427177492261, lbClusterId=3Dnull, networkPartit= ionId=3DN1] Cause: error running 1 node group(lb01lb01) location(RegionOne)= image(0299be4d-a743-4424-ae28-f40bd4faa669) size(2e2e2b47-9f40-4bd8-9777-8= 3e802f5f1cd) options({inboundPorts=3D[], autoAssignFloatingIp=3Dfalse, secu= rityGroupNames=3D[default], keyPairName=3Dphoenix, userData=3D[B@14edd531, = configDrive=3Dfalse, novaNetworks=3D[Network{networkUuid=3D42c4a88d-0d59-4f= bb-90f0-9b9806f9c17c, portUuid=3Dnull, fixedIp=3D172.16.2.201}, Network{net= workUuid=3D6a2615e4-760c-4c93-895d-b4b16e550193, portUuid=3Dnull, fixedIp= =3D10.81.69.201}, Network{networkUuid=3D670550f0-67fc-48ff-a33c-e184a790824= 7, portUuid=3Dnull, fixedIp=3D10.13.5.81}]}) > Execution failures: > 0 error[s]Node failures: > 1) IllegalStateException on node RegionOne/887351d5-2c16-48b9-927e-0ca5f1= 3fcc10:java.lang.IllegalStateException: node(RegionOne/887351d5-2c16-48b9-9= 27e-0ca5f13fcc10) didn't achieve the status running; aborting after 1 secon= ds with final status: ERROR > at org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRu= nning.java:72) > at org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRu= nning.java:45) at org.jclouds.compute.strategy.CustomizeNodeAndAddTo= GoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExcep= tionIntoBadMap.java:121) at org.jclouds.compute.strategy.CustomizeNo= deAndAddToGoodMapOrPutExceptionIntoBadMap.apply(CustomizeNodeAndAddToGoodMa= pOrPutExceptionIntoBadMap.java:146) at org.jclouds.compute.strategy.= CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.apply(CustomizeNodeAnd= AddToGoodMapOrPutExceptionIntoBadMap.java:53) > at com.google.common.util.concurrent.Futures$1.apply(Futures.java= :711) at com.google.common.util.concurrent.Futures$ChainingListenabl= eFuture.run(Futures.java:849) at java.util.concurrent.ThreadPoolExec= utor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent= .ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.= lang.Thread.run(Thread.java:745) > 1 error[s] > at org.apache.axis2.util.Utils.getInboundFaultFromMessageContext(= Utils.java:531) > at org.apache.axis2.description.OutInAxisOperationClient.handleRe= sponse(OutInAxisOperation.java:370) > at org.apache.axis2.description.OutInAxisOperationClient.send(Out= InAxisOperation.java:445) > at org.apache.axis2.description.OutInAxisOperationClient.executeI= mpl(OutInAxisOperation.java:225) > at org.apache.axis2.client.OperationClient.execute(OperationClien= t.java:149) > at org.apache.stratos.cloud.controller.stub.CloudControllerServic= eStub.startInstance(CloudControllerServiceStub.java:1407) at org.apa= che.stratos.autoscaler.client.cloud.controller.CloudControllerClient.spawnA= nInstance(CloudControllerClient.java:162) > ... 23 moreTID: [0] [STRATOS] [2015-03-24 06:13:04,511] INFO {or= g.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient} = - Trying to spawn an instance via cloud controller: [cluster] lb01.lb01.do= main [partition] RegionOne-Core [lb-cluster] null [network-partition-id] N1= {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClie= nt}TID: [0] [STRATOS] [2015-03-24 06:13:09,114] INFO {org.wso2.carbon.data= bridge.core.DataBridge} - admin connected {org.wso2.carbon.databridge.core= .DataBridge}TID: [0] [STRATOS] [2015-03-24 06:13:16,902] INFO {org.apache.= stratos.cloud.controller.impl.CloudControllerServiceImpl} - Instance is su= ccessfully starting up. MemberContext [memberId=3Dlb01.lb01.domain64df2bd7-= ee48-4ca2-9dec-362b72543d86, nodeId=3DRegionOne/aa1a0e56-a722-444a-99f5-080= ef844fb2d, clusterId=3Dlb01.lb01.domain, cartridgeType=3Dlb01, privateIpAdd= resses=3Dnull, publicIpAddresses=3Dnull, allocatedIpAddress=3Dnull, initTim= e=3D1427177584511, lbClusterId=3Dnull, networkPartitionId=3DN1] {org.apache= .stratos.cloud.controller.impl.CloudControllerServiceImpl} > at org.drools.common.AbstractWorkingMemory.fireAllRules(AbstractW= orkingMemory.java:674) > at org.drools.impl.StatefulKnowledgeSessionImpl.fireAllRules(Stat= efulKnowledgeSessionImpl.java:230) > at org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator.eva= luateMinCheck(AutoscalerRuleEvaluator.java:94) > at org.apache.stratos.autoscaler.monitor.ClusterMonitor.monitor(C= lusterMonitor.java:157) > at org.apache.stratos.autoscaler.monitor.ClusterMonitor.run(Clust= erMonitor.java:86) > at java.lang.Thread.run(Thread.java:745) > Caused by: org.apache.axis2.AxisFault: Failed to start an instance. Membe= rContext [memberId=3Dlb01.lb01.domaincde68af9-d82f-42c3-9c01-4fd8da565867, = nodeId=3Dnull, clusterId=3Dlb01.lb01.domain, cartridgeType=3Dlb01, privateI= pAddresses=3Dnull, publicIpAddresses=3Dnull, allocatedIpAddress=3Dnull, ini= tTime=3D1427177492261, lbClusterId=3Dnull, networkPartitionId=3DN1] Cause: = error running 1 node group(lb01lb01) location(RegionOne) image(0299be4d-a74= 3-4424-ae28-f40bd4faa669) size(2e2e2b47-9f40-4bd8-9777-83e802f5f1cd) option= s({inboundPorts=3D[], autoAssignFloatingIp=3Dfalse, securityGroupNames=3D[d= efault], keyPairName=3Dphoenix, userData=3D[B@14edd531, configDrive=3Dfalse= , novaNetworks=3D[Network{networkUuid=3D42c4a88d-0d59-4fbb-90f0-9b9806f9c17= c, portUuid=3Dnull, fixedIp=3D172.16.2.201}, Network{networkUuid=3D6a2615e4= -760c-4c93-895d-b4b16e550193, portUuid=3Dnull, fixedIp=3D10.81.69.201}, Net= work{networkUuid=3D670550f0-67fc-48ff-a33c-e184a7908247, portUuid=3Dnull, f= ixedIp=3D10.13.5.81}]}) > Execution failures: > 0 error[s] > Node failures: > 1) IllegalStateException on node RegionOne/887351d5-2c16-48b9-927e-0ca5f1= 3fcc10: > java.lang.IllegalStateException: node(RegionOne/887351d5-2c16-48b9-927e-0= ca5f13fcc10) didn't achieve the status running; aborting after 1 seconds wi= th final status: ERROR > at org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRu= nning.java:72) > at org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRu= nning.java:45) > at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPut= ExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadM= ap.java:121) > at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPut= ExceptionIntoBadMap.apply(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBad= Map.java:146) > at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPut= ExceptionIntoBadMap.apply(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBad= Map.java:53) > at com.google.common.util.concurrent.Futures$1.apply(Futures.java= :711) > at com.google.common.util.concurrent.Futures$ChainingListenableFu= ture.run(Futures.java:849) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolEx= ecutor.java:1145) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolE= xecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > 1 error[s] > at org.apache.axis2.util.Utils.getInboundFaultFromMessageContext(= Utils.java:531) > at org.apache.axis2.description.OutInAxisOperationClient.handleRe= sponse(OutInAxisOperation.java:370) > at org.apache.axis2.description.OutInAxisOperationClient.send(Out= InAxisOperation.java:445) > at org.apache.axis2.description.OutInAxisOperationClient.executeI= mpl(OutInAxisOperation.java:225) > at org.apache.axis2.client.OperationClient.execute(OperationClien= t.java:149) > at org.apache.stratos.cloud.controller.stub.CloudControllerServic= eStub.startInstance(CloudControllerServiceStub.java:1407) > at org.apache.stratos.autoscaler.client.cloud.controller.CloudCon= trollerClient.spawnAnInstance(CloudControllerClient.java:162) > ... 23 moreTID: [0] [STRATOS] [2015-03-24 06:11:34,509] ERROR {or= g.apache.stratos.autoscaler.monitor.ClusterMonitor} - Cluster monitor: Mon= itor failed.ClusterMonitor [clusterId=3Dlb01.lb01.domain, serviceId=3Dlb01,= deploymentPolicy=3DDeployment Policy [id]static-1-Core [partitions] [org.a= pache.stratos.cloud.controller.stub.deployment.partition.Partition@fb6144],= autoscalePolicy=3DASPolicy [id=3DeconomyPolicy, displayName=3Dnull, descri= ption=3Dnull], lbReferenceType=3Dnull, hasPrimary=3Dfalse ] {org.apache.str= atos.autoscaler.monitor.ClusterMonitor}Exception executing consequence for = rule "Minimum Rule" in org.apache.stratos.autoscaler.rule: java.lang.Runtim= eException: cannot invoke method: delegateSpawn at org.drools.runtim= e.rule.impl.DefaultConsequenceExceptionHandler.handleException(DefaultConse= quenceExceptionHandler.java:39) > at org.drools.common.DefaultAgenda.fireActivation(DefaultAgenda.j= ava:1297) > at org.drools.common.DefaultAgenda.fireNextItem(DefaultAgenda.jav= a:1221) > at org.drools.common.DefaultAgenda.fireAllRules(DefaultAgenda.jav= a:1456) > at org.drools.common.AbstractWorkingMemory.fireAllRules(AbstractW= orkingMemory.java:710) > at org.drools.common.AbstractWorkingMemory.fireAllRules(AbstractW= orkingMemory.java:674) > at org.drools.impl.StatefulKnowledgeSessionImpl.fireAllRules(Stat= efulKnowledgeSessionImpl.java:230) > at org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator.eva= luateMinCheck(AutoscalerRuleEvaluator.java:94) > at org.apache.stratos.autoscaler.monitor.ClusterMonitor.monitor(C= lusterMonitor.java:157) > at org.apache.stratos.autoscaler.monitor.ClusterMonitor.run(Clust= erMonitor.java:86) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.RuntimeException: cannot invoke method: delegateSpaw= n > at org.mvel2.optimizers.impl.refl.nodes.MethodAccessor.getValue(M= ethodAccessor.java:63) > at org.mvel2.optimizers.impl.refl.nodes.VariableAccessor.getValue= (VariableAccessor.java:37) > at org.mvel2.ast.ASTNode.getReducedValueAccelerated(ASTNode.java:= 108) at org.mvel2.MVELRuntime.execute(MVELRuntime.java:85) > at org.mvel2.compiler.CompiledExpression.getDirectValue(CompiledE= xpression.java:123) > at org.mvel2.compiler.CompiledExpression.getValue(CompiledExpress= ion.java:119) at org.mvel2.MVEL.executeExpression(MVEL.java:930) = at org.drools.base.mvel.MVELConsequence.evaluate(MVELConsequence.java:1= 04) at org.drools.common.DefaultAgenda.fireActivation(DefaultAgenda.= java:1287) > ... 9 moreCaused by: java.lang.reflect.InvocationTargetException > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) = at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImp= l.java:57) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMeth= odAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.mvel2.optimizers.impl.refl.nodes.MethodAccessor.getValue(M= ethodAccessor.java:48) > ... 17 more > Caused by: java.lang.RuntimeException: Cannot spawn an instance > at org.apache.stratos.autoscaler.rule.RuleTasksDelegator.delegate= Spawn(RuleTasksDelegator.java:107) > ... 22 moreCaused by: org.apache.stratos.autoscaler.exception.Spa= wningException: Failed to start an instance. MemberContext [memberId=3Dlb01= .lb01.domaincde68af9-d82f-42c3-9c01-4fd8da565867, nodeId=3Dnull, clusterId= =3Dlb01.lb01.domain, cartridgeType=3Dlb01, privateIpAddresses=3Dnull, publi= cIpAddresses=3Dnull, allocatedIpAddress=3Dnull, initTime=3D1427177492261, l= bClusterId=3Dnull, networkPartitionId=3DN1] Cause: error running 1 node gro= up(lb01lb01) location(RegionOne) image(0299be4d-a743-4424-ae28-f40bd4faa669= ) size(2e2e2b47-9f40-4bd8-9777-83e802f5f1cd) options({inboundPorts=3D[], au= toAssignFloatingIp=3Dfalse, securityGroupNames=3D[default], keyPairName=3Dp= hoenix, userData=3D[B@14edd531, configDrive=3Dfalse, novaNetworks=3D[Networ= k{networkUuid=3D42c4a88d-0d59-4fbb-90f0-9b9806f9c17c, portUuid=3Dnull, fixe= dIp=3D172.16.2.201}, Network{networkUuid=3D6a2615e4-760c-4c93-895d-b4b16e55= 0193, portUuid=3Dnull, fixedIp=3D10.81.69.201}, Network{networkUuid=3D67055= 0f0-67fc-48ff-a33c-e184a7908247, portUuid=3Dnull, fixedIp=3D10.13.5.81}]}) > Execution failures: > 0 error[s] > Node failures: > 1) IllegalStateException on node RegionOne/887351d5-2c16-48b9-927e-0ca5f1= 3fcc10:java.lang.IllegalStateException: node(RegionOne/887351d5-2c16-48b9-9= 27e-0ca5f13fcc10) didn't achieve the status running; aborting after 1 secon= ds with final status: ERROR > at org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRu= nning.java:72) > at org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRu= nning.java:45) at org.jclouds.compute.strategy.CustomizeNodeAndAddTo= GoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExcep= tionIntoBadMap.java:121) at org.jclouds.compute.strategy.CustomizeNo= deAndAddToGoodMapOrPutExceptionIntoBadMap.apply(CustomizeNodeAndAddToGoodMa= pOrPutExceptionIntoBadMap.java:146) at org.jclouds.compute.strategy.= CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.apply(CustomizeNodeAnd= AddToGoodMapOrPutExceptionIntoBadMap.java:53) > at com.google.common.util.concurrent.Futures$1.apply(Futures.java= :711) at com.google.common.util.concurrent.Futures$ChainingListenabl= eFuture.run(Futures.java:849) at java.util.concurrent.ThreadPoolExec= utor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent= .ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.= lang.Thread.run(Thread.java:745) > 1 error[s] at org.apache.stratos.autoscaler.client.cloud.controlle= r.CloudControllerClient.spawnAnInstance(CloudControllerClient.java:174) > at org.apache.stratos.autoscaler.rule.RuleTasksDelegator.delegate= Spawn(RuleTasksDelegator.java:87) > ... 22 moreCaused by: org.apache.axis2.AxisFault: Failed to start= an instance. MemberContext [memberId=3Dlb01.lb01.domaincde68af9-d82f-42c3-= 9c01-4fd8da565867, nodeId=3Dnull, clusterId=3Dlb01.lb01.domain, cartridgeTy= pe=3Dlb01, privateIpAddresses=3Dnull, publicIpAddresses=3Dnull, allocatedIp= Address=3Dnull, initTime=3D1427177492261, lbClusterId=3Dnull, networkPartit= ionId=3DN1] Cause: error running 1 node group(lb01lb01) location(RegionOne)= image(0299be4d-a743-4424-ae28-f40bd4faa669) size(2e2e2b47-9f40-4bd8-9777-8= 3e802f5f1cd) options({inboundPorts=3D[], autoAssignFloatingIp=3Dfalse, secu= rityGroupNames=3D[default], keyPairName=3Dphoenix, userData=3D[B@14edd531, = configDrive=3Dfalse, novaNetworks=3D[Network{networkUuid=3D42c4a88d-0d59-4f= bb-90f0-9b9806f9c17c, portUuid=3Dnull, fixedIp=3D172.16.2.201}, Network{net= workUuid=3D6a2615e4-760c-4c93-895d-b4b16e550193, portUuid=3Dnull, fixedIp= =3D10.81.69.201}, Network{networkUuid=3D670550f0-67fc-48ff-a33c-e184a790824= 7, portUuid=3Dnull, fixedIp=3D10.13.5.81}]}) > Execution failures: > 0 error[s]Node failures: > 1) IllegalStateException on node RegionOne/887351d5-2c16-48b9-927e-0ca5f1= 3fcc10:java.lang.IllegalStateException: node(RegionOne/887351d5-2c16-48b9-9= 27e-0ca5f13fcc10) didn't achieve the status running; aborting after 1 secon= ds with final status: ERROR > at org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRu= nning.java:72) > at org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRu= nning.java:45) at org.jclouds.compute.strategy.CustomizeNodeAndAddTo= GoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExcep= tionIntoBadMap.java:121) at org.jclouds.compute.strategy.CustomizeNo= deAndAddToGoodMapOrPutExceptionIntoBadMap.apply(CustomizeNodeAndAddToGoodMa= pOrPutExceptionIntoBadMap.java:146) at org.jclouds.compute.strategy.= CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.apply(CustomizeNodeAnd= AddToGoodMapOrPutExceptionIntoBadMap.java:53) > at com.google.common.util.concurrent.Futures$1.apply(Futures.java= :711) at com.google.common.util.concurrent.Futures$ChainingListenabl= eFuture.run(Futures.java:849) at java.util.concurrent.ThreadPoolExec= utor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent= .ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.= lang.Thread.run(Thread.java:745) > 1 error[s] > at org.apache.axis2.util.Utils.getInboundFaultFromMessageContext(= Utils.java:531) > at org.apache.axis2.description.OutInAxisOperationClient.handleRe= sponse(OutInAxisOperation.java:370) > at org.apache.axis2.description.OutInAxisOperationClient.send(Out= InAxisOperation.java:445) > at org.apache.axis2.description.OutInAxisOperationClient.executeI= mpl(OutInAxisOperation.java:225) > at org.apache.axis2.client.OperationClient.execute(OperationClien= t.java:149) > at org.apache.stratos.cloud.controller.stub.CloudControllerServic= eStub.startInstance(CloudControllerServiceStub.java:1407) at org.apa= che.stratos.autoscaler.client.cloud.controller.CloudControllerClient.spawnA= nInstance(CloudControllerClient.java:162) > ... 23 more > TID: [0] [STRATOS] [2015-03-24 06:13:04,511] INFO {org.apache.stratos.au= toscaler.client.cloud.controller.CloudControllerClient} - Trying to spawn = an instance via cloud controller: [cluster] lb01.lb01.domain [partition] Re= gionOne-Core [lb-cluster] null [network-partition-id] N1 {org.apache.strato= s.autoscaler.client.cloud.controller.CloudControllerClient}TID: [0] [STRATO= S] [2015-03-24 06:13:09,114] INFO {org.wso2.carbon.databridge.core.DataBri= dge} - admin connected {org.wso2.carbon.databridge.core.DataBridge}TID: [0= ] [STRATOS] [2015-03-24 06:13:16,902] INFO {org.apache.stratos.cloud.contr= oller.impl.CloudControllerServiceImpl} - Instance is successfully starting= up. MemberContext [memberId=3Dlb01.lb01.domain64df2bd7-ee48-4ca2-9dec-362b= 72543d86, nodeId=3DRegionOne/aa1a0e56-a722-444a-99f5-080ef844fb2d, clusterI= d=3Dlb01.lb01.domain, cartridgeType=3Dlb01, privateIpAddresses=3Dnull, publ= icIpAddresses=3Dnull, allocatedIpAddress=3Dnull, initTime=3D1427177584511, = lbClusterId=3Dnull, networkPartitionId=3DN1] {org.apache.stratos.cloud.cont= roller.impl.CloudControllerServiceImpl} -- This message was sent by Atlassian JIRA (v6.3.4#6332)