Return-Path: X-Original-To: apmail-cloudstack-users-archive@www.apache.org Delivered-To: apmail-cloudstack-users-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id D2BA911FDF for ; Wed, 17 Sep 2014 04:17:40 +0000 (UTC) Received: (qmail 89703 invoked by uid 500); 17 Sep 2014 04:17:39 -0000 Delivered-To: apmail-cloudstack-users-archive@cloudstack.apache.org Received: (qmail 89654 invoked by uid 500); 17 Sep 2014 04:17:39 -0000 Mailing-List: contact users-help@cloudstack.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: users@cloudstack.apache.org Delivered-To: mailing list users@cloudstack.apache.org Received: (qmail 89638 invoked by uid 99); 17 Sep 2014 04:17:39 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 17 Sep 2014 04:17:39 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=5.0 tests=SPF_HELO_PASS,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of cloudstck@trick-solutions.com designates 216.249.106.136 as permitted sender) Received: from [216.249.106.136] (HELO mail.w3fc.com) (216.249.106.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 17 Sep 2014 04:17:11 +0000 Received: from localhost (localhost [127.0.0.1]) by mail.w3fc.com (Postfix) with ESMTP id 5845CB4273F for ; Wed, 17 Sep 2014 00:17:09 -0400 (EDT) X-Virus-Scanned: Debian amavisd-new at mail.w3fc.com Received: from mail.w3fc.com ([216.249.106.136]) by localhost (web1.w3fc.com [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id A5sLyb5CMSGi for ; Wed, 17 Sep 2014 00:17:04 -0400 (EDT) Received: from Tek505 (cpe-174-096-227-221.carolina.res.rr.com [174.96.227.221]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: cloudstck@trick-solutions.com) by mail.w3fc.com (Postfix) with ESMTPSA id C7480B4229D for ; Wed, 17 Sep 2014 00:17:03 -0400 (EDT) From: "Matthew Midgett" To: References: <53DA552C.5010909@procne.it> ,<53DB4A66.1050309@procne.it> <94B6896A-874A-4D65-B7A7-734726E2DB77@shapeblue.com>,<54114C73.4040906@procne.it> <152B12C7-7947-4A33-A7F9-33ACDB87FBE8@shapeblue.com> <54180465.7010409@procne.it> <007b01cfd217$1a00bb10$4e023130$@trick-solutions.com> <5418FDB8.5010202@gmail.com> In-Reply-To: <5418FDB8.5010202@gmail.com> Subject: RE: Unable to start a VM due to insufficient capacity Date: Wed, 17 Sep 2014 00:17:02 -0400 Message-ID: <008f01cfd22e$3c106630$b4313290$@trick-solutions.com> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Mailer: Microsoft Outlook 15.0 Content-Language: en-us Thread-Index: AQLuhETVVOjbWFrR/W6+zobz7j+mqAIIG5tvAbn8j3cBhbHsCgJ/niv9AdsfYYoByL/DuAKrSDDBAxflQf2ZPdDxwA== X-Virus-Checked: Checked by ClamAV on apache.org Thank you for your answer and I will try that. -----Original Message----- From: ilya musayev [mailto:ilya.mailing.lists@gmail.com] Sent: Tuesday, September 16, 2014 11:19 PM To: users@cloudstack.apache.org Subject: Re: Unable to start a VM due to insufficient capacity I've seen this before and had similar issues. Instead of re-installing or dropping DB, you can delete all job_ids that are stuck. For example, this is what i would see in my logs. (1013,VmWorkJobQueue, 2375) is reaching concurrency limit 1 (1013,VmWorkJobQueue, 2375) is reaching concurrency limit 1 (1433,VmWorkJobQueue, 2470) is reaching concurrency limit 1 (1434,VmWorkJobQueue, 2471) is reaching concurrency limit 1 (1475,VmWorkJobQueue, 2483) is reaching concurrency limit 1 (1477,VmWorkJobQueue, 2484) is reaching concurrency limit 1 (1478,VmWorkJobQueue, 596) is reaching concurrency limit 1 (981,VmWorkJobQueue, 2371) is reaching concurrency limit 1 DELETE ALL FROM BELOW: SELECT * FROM cloud.sync_queue_item where queue_id=981 or queue_id=1478 or queue_id=1477 or queue_id=1475 or queue_id=1434 or queue_id=1433 or queue_id=1013 ; SELECT * FROM cloud.sync_queue where id=1433 or id=1434 or id=1475 or id=1477 or id=1478 or id=981; While i did SELECT statement, you can alter to DELETE. Regards ilya On 9/16/14, 6:31 PM, Matthew Midgett wrote: > Ugo, I am having an issue just like you. I can deploy the default > centos template but if I try to deploy a iso it looks like its failing > in the network area. I've been reading your emails to Geoff and I have > restarted the network with a clean and that didn't fix my problem. I > also restarted the management servers during this process. I don't > know what happed but ssvm and console quit working. I've spent all day > deleting the whole thing and setting it back up again. Just to find that it's not working. > > Now I just saw this in the logs > > Cannot delete or update a parent row: a foreign key constraint fails > (`cloud`.`async_job_join_map`, CONSTRAINT > `fk_async_job_join_map__join_job_id` FOREIGN KEY (`join_job_id`) > REFERENCES `async_job` (`id`)) > > I am running a 2 node haproxy with virtual ip in front of a galera sql > cluster. > > I dropped the DB, gave the cloud user more permissions. Deleted > everything and restarted and it's doing this all over again. I am > running xenserver 6.2 and acs4.4. I even tried 4.3 again and it started doing the same thing. > > > > -----Original Message----- > From: Ugo Vasi [mailto:ugo.vasi@procne.it] > Sent: Tuesday, September 16, 2014 5:36 AM > To: users@cloudstack.apache.org > Subject: Re: Unable to start a VM due to insufficient capacity > > Hi Geoff, > I restarted the network and now I can create new VM without errors. > > Now, how can I prevent this issue? > > > > > Il 11/09/2014 09:28, Geoff Higginbottom ha scritto: >> Hi Ugo, >> >> Did restarting the network with cleanup option resolve your issue >> last > time, if yes have you tried the same on this network? >> Regards >> >> Geoff Higginbottom >> CTO / Cloud Architect >> >> D: +44 20 3603 0542 | S: +44 20 3603 >> 0540 | M: +447968161581 >> >> geoff.higginbottom@shapeblue.com> c >> om> | www.shapeblue.com | >> Twitter:@cloudstackguru >> >> ShapeBlue Ltd, 53 Chandos Place, Covent Garden, London, WC2N >> 4HS >> >> >> On 11 Sep 2014, at 08:17, "Ugo Vasi" > > wrote: >> Hi Geoff, >> I still have the problem creating new VMs but in another isolated network. > The problem seems to be the same: >> 2014-09-11 08:55:27,048 DEBUG [c.c.v.VirtualMachineManagerImpl] >> (Job-Executor-55:ctx-b5c34ec7 ctx-3715f5de) VM is being created in >> podId: 1 >> 2014-09-11 08:55:27,050 DEBUG [o.a.c.e.o.NetworkOrchestrator] >> (Job-Executor-55:ctx-b5c34ec7 ctx-3715f5de) Network id=212 is already >> implemented >> 2014-09-11 08:55:27,118 DEBUG [c.c.n.NetworkModelImpl] >> (Job-Executor-55:ctx-b5c34ec7 ctx-3715f5de) Service SecurityGroup is >> not supported in the network id=212 >> 2014-09-11 08:55:27,122 DEBUG [o.a.c.e.o.NetworkOrchestrator] >> (Job-Executor-55:ctx-b5c34ec7 ctx-3715f5de) Changing active number of >> nics for network id=212 on 1 >> 2014-09-11 08:55:27,174 DEBUG [o.a.c.e.o.NetworkOrchestrator] >> (Job-Executor-55:ctx-b5c34ec7 ctx-3715f5de) Asking VirtualRouter to >> prepare for >> Nic[116-86-8429f13f-7c3b-4b8a-aea3-9b8e9a3f4c03-10.1.1.19] >> 2014-09-11 08:55:27,181 DEBUG >> [c.c.n.r.VirtualNetworkApplianceManagerImpl] >> (Job-Executor-55:ctx-b5c34ec7 ctx-3715f5de) Lock is acquired for >> network id 212 as a part of router startup in >> Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->P >> o >> ol(Id))] : >> Dest[Zone(1)-Pod(1)-Cluster(1)-Host(7)-Storage(Volume(98|ROOT-->Pool( >> 6 >> ))] >> 2014-09-11 08:55:27,183 DEBUG >> [c.c.n.r.VirtualNetworkApplianceManagerImpl] >> (Job-Executor-55:ctx-b5c34ec7 ctx-3715f5de) Lock is released for >> network id 212 as a part of router startup in >> Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->P >> o >> ol(Id))] : >> Dest[Zone(1)-Pod(1)-Cluster(1)-Host(7)-Storage(Volume(98|ROOT-->Pool( >> 6 >> ))] >> 2014-09-11 08:55:27,189 DEBUG [c.c.n.NetworkModelImpl] >> (Job-Executor-55:ctx-b5c34ec7 ctx-3715f5de) Service SecurityGroup is >> not supported in the network id=212 >> 2014-09-11 08:55:27,198 DEBUG [c.c.n.NetworkModelImpl] >> (Job-Executor-55:ctx-b5c34ec7 ctx-3715f5de) Service SecurityGroup is >> not supported in the network id=212 >> 2014-09-11 08:55:27,202 DEBUG >> [c.c.n.r.VirtualNetworkApplianceManagerImpl] >> (Job-Executor-55:ctx-b5c34ec7 ctx-3715f5de) Applying dhcp entry in >> network Ntwk[212|Guest|8] >> 2014-09-11 08:55:27,215 DEBUG [c.c.a.t.Request] >> (Job-Executor-55:ctx-b5c34ec7 ctx-3715f5de) Seq 7-564304056: Sending >> { Cmd , MgmtId: 77916390353436, via: 7(csh1b), Ver: v1, Flags: >> 100011, >> [{"com.cloud.agent.api.routing.DhcpEntryCommand":{"vmMac":"02:00:15:7 >> 6 >> :00:0e","vmIpAddress":"10.1.1.19","vmName":"giannitest","defaultRoute >> r >> ":"10.1.1.1","defaultDns":"10.1.1.1","duid":"00:03:00:01:02:00:15:76: >> 0 >> 0:0e","isDefault":true,"executeInSequence":false,"accessDetails":{"ro >> u >> ter.guest.ip":"10.1.1.1","zone.network.type":"Advanced","router.name": >> "r-60-VM","router.ip":"169.254.1.230"},"wait":0}}] } >> 2014-09-11 08:55:27,941 DEBUG [c.c.s.s.SecondaryStorageManagerImpl] >> (secstorage-1:ctx-327037b2) Zone 1 is ready to launch secondary >> storage VM >> 2014-09-11 08:55:28,022 DEBUG [c.c.c.ConsoleProxyManagerImpl] >> (consoleproxy-1:ctx-a8ba3b1e) Zone 1 is ready to launch console proxy >> 2014-09-11 08:55:28,820 DEBUG [c.c.a.ApiServlet] >> (catalina-exec-16:ctx-58e92382) ===START=== 192.168.130.89 -- GET >> command=queryAsyncJobResult&jobId=195bf80a-9823-4ec3-add4-8907b98fbaf >> 0 >> &response=json&sessionkey=qGZA6KYoc1SB%2Bq2k70HoM%2BBBQTU%3D&_=141041 >> 8 >> 528731 >> 2014-09-11 08:55:29,069 DEBUG [c.c.a.t.Request] >> (AgentManager-Handler-13:null) Seq 7-564304056: Processing: { Ans: , >> MgmtId: 77916390353436, via: 7, Ver: v1, Flags: 10, >> [{"com.cloud.agent.api.Answer":{"result":true,"wait":0}}] } >> 2014-09-11 08:55:29,069 DEBUG [c.c.a.t.Request] >> (Job-Executor-55:ctx-b5c34ec7 ctx-3715f5de) Seq 7-564304056: Received: >> { Ans: , MgmtId: 77916390353436, via: 7, Ver: v1, Flags: 10, { Answer >> } } >> 2014-09-11 08:55:29,075 DEBUG [c.c.n.NetworkModelImpl] >> (Job-Executor-55:ctx-b5c34ec7 ctx-3715f5de) Service SecurityGroup is >> not supported in the network id=212 >> 2014-09-11 08:55:29,078 DEBUG >> [c.c.n.r.VirtualNetworkApplianceManagerImpl] >> (Job-Executor-55:ctx-b5c34ec7 ctx-3715f5de) Applying userdata and >> password entry in network Ntwk[212|Guest|8] >> 2014-09-11 08:55:29,089 DEBUG [c.c.a.t.Request] >> (Job-Executor-55:ctx-b5c34ec7 ctx-3715f5de) Seq 7-564304057: Sending >> { Cmd , MgmtId: 77916390353436, via: 7(csh1b), Ver: v1, Flags: >> 100011, >> [{"com.cloud.agent.api.routing.SavePasswordCommand":{,"vmIpAddress":" >> 1 >> 0.1.1.19","vmName":"giannitest","executeInSequence":false,"accessDeta >> i >> ls":{"router.guest.ip":"10.1.1.1","zone.network.type":"Advanced","rou >> t >> er.ip":"169.254.1.230","router.name":"r-60-VM"},"wait":0}},{"com.clou >> d >> .agent.api.routing.VmDataCommand":{"vmIpAddress":"10.1.1.19","vmName": >> "giannitest","executeInSequence":false,"accessDetails":{"router.guest. >> ip":"10.1.1.1","zone.network.type":"Advanced","router.ip":"169.254.1. >> 2 30","router.name":"r-60-VM"},"wait":0}}] } >> 2014-09-11 08:55:29,399 DEBUG [c.c.a.t.Request] >> (AgentManager-Handler-10:null) Seq 7-564304057: Processing: { Ans: , >> MgmtId: 77916390353436, via: 7, Ver: v1, Flags: 10, >> [{"com.cloud.agent.api.Answer":{"result":true,"wait":0}},{"com.cloud. >> a gent.api.Answer":{"result":false,"details":"VmDataCommand failed, >> check agent logs","wait":0}}] } >> 2014-09-11 08:55:29,491 DEBUG [c.c.a.t.Request] >> (Job-Executor-55:ctx-b5c34ec7 ctx-3715f5de) Seq 7-564304057: Received: >> { Ans: , MgmtId: 77916390353436, via: 7, Ver: v1, Flags: 10, { >> Answer, Answer } } >> 2014-09-11 08:55:29,491 INFO [c.c.v.VirtualMachineManagerImpl] > (Job-Executor-55:ctx-b5c34ec7 ctx-3715f5de) Unable to contact resource. >> com.cloud.exception.ResourceUnavailableException: Resource >> [DataCenter:1] > is unreachable: Unable to apply userdata and password entry on router >> at > com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyRules > (Virtu > alNetworkApplianceManagerImpl.java:3915) >> at > com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyUserD > ata(Vi > rtualNetworkApplianceManagerImpl.java:3118) >> at sun.reflect.GeneratedMethodAccessor472.invoke(Unknown >> Source) ... >> 2014-09-11 08:55:29,755 DEBUG [c.c.a.ApiServlet] >> (catalina-exec-16:ctx-58e92382 ctx-ad8ddf67) ===END=== 192.168.130.89 >> -- GET >> command=queryAsyncJobResult&jobId=195bf80a-9823-4ec3-add4-8907b98fbaf >> 0 >> &response=json&sessionkey=qGZA6KYoc1SB%2Bq2k70HoM%2BBBQTU%3D&_=141041 >> 8 >> 528731 >> 2014-09-11 08:55:29,798 DEBUG [c.c.v.VirtualMachineManagerImpl] >> (Job-Executor-55:ctx-b5c34ec7 ctx-3715f5de) Cleaning up resources for >> the vm VM[User|giannitest] in Starting state >> 2014-09-11 08:55:29,800 DEBUG [c.c.a.t.Request] >> (Job-Executor-55:ctx-b5c34ec7 ctx-3715f5de) Seq 7-564304058: Sending >> { Cmd , MgmtId: 77916390353436, via: 7(csh1b), Ver: v1, Flags: >> 100011, >> [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequen >> c e":false,"vmName":"i-2-86-VM","wait":0}}] } >> 2014-09-11 08:55:29,950 DEBUG [c.c.a.t.Request] >> (AgentManager-Handler-6:null) Seq 7-564304058: Processing: { Ans: , >> MgmtId: 77916390353436, via: 7, Ver: v1, Flags: 10, >> [{"com.cloud.agent.api.StopAnswer":{"result":true,"wait":0}}] } >> 2014-09-11 08:55:29,950 DEBUG [c.c.a.t.Request] >> (Job-Executor-55:ctx-b5c34ec7 ctx-3715f5de) Seq 7-564304058: Received: >> { Ans: , MgmtId: 77916390353436, via: 7, Ver: v1, Flags: 10, { >> StopAnswer } } >> 2014-09-11 08:55:29,955 DEBUG [c.c.n.NetworkModelImpl] >> (Job-Executor-55:ctx-b5c34ec7 ctx-3715f5de) Service SecurityGroup is >> not supported in the network id=212 >> 2014-09-11 08:55:29,957 DEBUG [o.a.c.e.o.NetworkOrchestrator] >> (Job-Executor-55:ctx-b5c34ec7 ctx-3715f5de) Changing active number of >> nics for network id=212 on -1 >> 2014-09-11 08:55:30,051 DEBUG [o.a.c.e.o.NetworkOrchestrator] >> (Job-Executor-55:ctx-b5c34ec7 ctx-3715f5de) Asking VirtualRouter to >> release >> NicProfile[116-86-8429f13f-7c3b-4b8a-aea3-9b8e9a3f4c03-10.1.1.19-null >> 2014-09-11 08:55:30,051 DEBUG [c.c.v.VirtualMachineManagerImpl] >> (Job-Executor-55:ctx-b5c34ec7 ctx-3715f5de) Successfully released >> network resources for the vm VM[User|giannitest] >> 2014-09-11 08:55:30,051 DEBUG [c.c.v.VirtualMachineManagerImpl] >> (Job-Executor-55:ctx-b5c34ec7 ctx-3715f5de) Successfully cleanued up >> resources for the vm VM[User|giannitest] in Starting state >> 2014-09-11 08:55:30,058 DEBUG [c.c.d.DeploymentPlanningManagerImpl] >> (Job-Executor-55:ctx-b5c34ec7 ctx-3715f5de) Deploy avoids pods: [], >> clusters: [], hosts: [7] >> 2014-09-11 08:55:30,058 DEBUG [c.c.d.DeploymentPlanningManagerImpl] > (Job-Executor-55:ctx-b5c34ec7 ctx-3715f5de) DataCenter id = '1' > provided is in avoid set, DeploymentPlanner cannot allocate the VM, returning. >> 2014-09-11 08:55:30,214 DEBUG [c.c.c.CapacityManagerImpl] >> (Job-Executor-55:ctx-b5c34ec7 ctx-3715f5de) VM state transitted from >> :Starting to Stopped with event: OperationFailedvm's original host id: >> null new host id: null host id before state transition: 7 >> 2014-09-11 08:55:30,218 DEBUG [c.c.c.CapacityManagerImpl] >> (Job-Executor-55:ctx-b5c34ec7 ctx-3715f5de) Hosts's actual total CPU: >> 36256 and CPU after applying overprovisioning: 36256 >> 2014-09-11 08:55:30,218 DEBUG [c.c.c.CapacityManagerImpl] >> (Job-Executor-55:ctx-b5c34ec7 ctx-3715f5de) Hosts's actual total RAM: >> 50639785984 and RAM after applying overprovisioning: 50639785984 >> 2014-09-11 08:55:30,218 DEBUG [c.c.c.CapacityManagerImpl] >> (Job-Executor-55:ctx-b5c34ec7 ctx-3715f5de) release cpu from host: 7, >> old used: 26500,reserved: 0, actual total: 36256, total with >> overprovisioning: 36256; new used: 26000,reserved:0; >> movedfromreserved: false,moveToReserveredfalse >> 2014-09-11 08:55:30,218 DEBUG [c.c.c.CapacityManagerImpl] >> (Job-Executor-55:ctx-b5c34ec7 ctx-3715f5de) release mem from host: 7, >> old used: 21072183296,reserved: 0, total: 50639785984; new used: >> 20535312384,reserved:0; movedfromreserved: >> false,moveToReserveredfalse >> 2014-09-11 08:55:30,302 DEBUG [c.c.c.CapacityManagerImpl] >> (Job-Executor-55:ctx-b5c34ec7 ctx-3715f5de) VM state transitted from >> :Stopped to Starting with event: StartRequestedvm's original host id: >> null new host id: null host id before state transition: null >> 2014-09-11 08:55:30,302 DEBUG [c.c.v.VirtualMachineManagerImpl] >> (Job-Executor-55:ctx-b5c34ec7 ctx-3715f5de) Successfully transitioned >> to start state for VM[User|giannitest] reservation id = >> 4b6c2f8d-a34c-4b86-a0d4-acd5ea87353a >> 2014-09-11 08:55:30,340 DEBUG [c.c.v.VirtualMachineManagerImpl] >> (Job-Executor-55:ctx-b5c34ec7 ctx-3715f5de) Trying to deploy VM, vm >> has dcId: 1 and podId: 1 >> 2014-09-11 08:55:30,340 DEBUG [c.c.v.VirtualMachineManagerImpl] >> (Job-Executor-55:ctx-b5c34ec7 ctx-3715f5de) Deploy avoids pods: [], >> clusters: [], hosts: [7] >> 2014-09-11 08:55:30,348 DEBUG [c.c.d.DeploymentPlanningManagerImpl] >> (Job-Executor-55:ctx-b5c34ec7 ctx-3715f5de) Deploy avoids pods: [], >> clusters: [], hosts: [7] >> 2014-09-11 08:55:30,348 DEBUG [c.c.d.DeploymentPlanningManagerImpl] > (Job-Executor-55:ctx-b5c34ec7 ctx-3715f5de) DataCenter id = '1' > provided is in avoid set, DeploymentPlanner cannot allocate the VM, returning. >> 2014-09-11 08:55:30,435 DEBUG [c.c.c.CapacityManagerImpl] >> (Job-Executor-55:ctx-b5c34ec7 ctx-3715f5de) VM state transitted from >> :Starting to Stopped with event: OperationFailedvm's original host id: >> null new host id: null host id before state transition: null >> 2014-09-11 08:55:30,477 DEBUG [c.c.v.UserVmManagerImpl] >> (Job-Executor-55:ctx-b5c34ec7 ctx-3715f5de) Destroying vm >> VM[User|giannitest] as it failed to create on Host with Id:null >> 2014-09-11 08:55:30,609 DEBUG [c.c.c.CapacityManagerImpl] >> (Job-Executor-55:ctx-b5c34ec7 ctx-3715f5de) VM state transitted from >> :Stopped to Error with event: OperationFailedToErrorvm's original >> host >> id: null new host id: null host id before state transition: null >> 2014-09-11 08:55:29,755 DEBUG [c.c.a.ApiServlet] >> (catalina-exec-16:ctx-58e92382 ctx-ad8ddf67) ===END=== 192.168.130.89 >> -- GET >> command=queryAsyncJobResult&jobId=195bf80a-9823-4ec3-add4-8907b98fbaf >> 0 >> &response=json&sessionkey=qGZA6KYoc1SB%2Bq2k70HoM%2BBBQTU%3D&_=141041 >> 8 >> 528731 >> 2014-09-11 08:55:29,798 DEBUG [c.c.v.VirtualMachineManagerImpl] >> (Job-Executor-55:ctx-b5c34ec7 ctx-3715f5de) Cleaning up resources for >> the vm VM[User|giannitest] in Starting state >> 2014-09-11 08:55:29,800 DEBUG [c.c.a.t.Request] >> (Job-Executor-55:ctx-b5c34ec7 ctx-3715f5de) Seq 7-564304058: Sending >> { Cmd , MgmtId: 77916390353436, via: 7(csh1b), Ver: v1, Flags: >> 100011, >> [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequen >> c e":false,"vmName":"i-2-86-VM","wait":0}}] } >> 2014-09-11 08:55:29,950 DEBUG [c.c.a.t.Request] >> (AgentManager-Handler-6:null) Seq 7-564304058: Processing: { Ans: , >> MgmtId: 77916390353436, via: 7, Ver: v1, Flags: 10, >> [{"com.cloud.agent.api.StopAnswer":{"result":true,"wait":0}}] } >> 2014-09-11 08:55:29,950 DEBUG [c.c.a.t.Request] >> (Job-Executor-55:ctx-b5c34ec7 ctx-3715f5de) Seq 7-564304058: Received: >> { Ans: , MgmtId: 77916390353436, via: 7, Ver: v1, Flags: 10, { >> StopAnswer } } >> 2014-09-11 08:55:29,955 DEBUG [c.c.n.NetworkModelImpl] >> (Job-Executor-55:ctx-b5c34ec7 ctx-3715f5de) Service SecurityGroup is >> not supported in the network id=212 >> 2014-09-11 08:55:29,957 DEBUG [o.a.c.e.o.NetworkOrchestrator] >> (Job-Executor-55:ctx-b5c34ec7 ctx-3715f5de) Changing active number of >> nics for network id=212 on -1 >> 2014-09-11 08:55:30,051 DEBUG [o.a.c.e.o.NetworkOrchestrator] >> (Job-Executor-55:ctx-b5c34ec7 ctx-3715f5de) Asking VirtualRouter to >> release >> NicProfile[116-86-8429f13f-7c3b-4b8a-aea3-9b8e9a3f4c03-10.1.1.19-null >> 2014-09-11 08:55:30,051 DEBUG [c.c.v.VirtualMachineManagerImpl] >> (Job-Executor-55:ctx-b5c34ec7 ctx-3715f5de) Successfully released >> network resources for the vm VM[User|giannitest] >> 2014-09-11 08:55:30,051 DEBUG [c.c.v.VirtualMachineManagerImpl] >> (Job-Executor-55:ctx-b5c34ec7 ctx-3715f5de) Successfully cleanued up >> resources for the vm VM[User|giannitest] in Starting state >> 2014-09-11 08:55:30,058 DEBUG [c.c.d.DeploymentPlanningManagerImpl] >> (Job-Executor-55:ctx-b5c34ec7 ctx-3715f5de) Deploy avoids pods: [], >> clusters: [], hosts: [7] >> 2014-09-11 08:55:30,058 DEBUG [c.c.d.DeploymentPlanningManagerImpl] > (Job-Executor-55:ctx-b5c34ec7 ctx-3715f5de) DataCenter id = '1' > provided is in avoid set, DeploymentPlanner cannot allocate the VM, returning. >> 2014-09-11 08:55:30,214 DEBUG [c.c.c.CapacityManagerImpl] >> (Job-Executor-55:ctx-b5c34ec7 ctx-3715f5de) VM state transitted from >> :Starting to Stopped with event: OperationFailedvm's original host id: >> null new host id: null host id before state transition: 7 >> 2014-09-11 08:55:30,218 DEBUG [c.c.c.CapacityManagerImpl] >> (Job-Executor-55:ctx-b5c34ec7 ctx-3715f5de) Hosts's actual total CPU: >> 36256 and CPU after applying overprovisioning: 36256 >> 2014-09-11 08:55:30,218 DEBUG [c.c.c.CapacityManagerImpl] >> (Job-Executor-55:ctx-b5c34ec7 ctx-3715f5de) Hosts's actual total RAM: >> 50639785984 and RAM after applying overprovisioning: 50639785984 >> 2014-09-11 08:55:30,218 DEBUG [c.c.c.CapacityManagerImpl] >> (Job-Executor-55:ctx-b5c34ec7 ctx-3715f5de) release cpu from host: 7, >> old used: 26500,reserved: 0, actual total: 36256, total with >> overprovisioning: 36256; new used: 26000,reserved:0; >> movedfromreserved: false,moveToReserveredfalse >> 2014-09-11 08:55:30,218 DEBUG [c.c.c.CapacityManagerImpl] >> (Job-Executor-55:ctx-b5c34ec7 ctx-3715f5de) release mem from host: 7, >> old used: 21072183296,reserved: 0, total: 50639785984; new used: >> 20535312384,reserved:0; movedfromreserved: >> false,moveToReserveredfalse >> 2014-09-11 08:55:30,302 DEBUG [c.c.c.CapacityManagerImpl] >> (Job-Executor-55:ctx-b5c34ec7 ctx-3715f5de) VM state transitted from >> :Stopped to Starting with event: StartRequestedvm's original host id: >> null new host id: null host id before state transition: null >> 2014-09-11 08:55:30,302 DEBUG [c.c.v.VirtualMachineManagerImpl] >> (Job-Executor-55:ctx-b5c34ec7 ctx-3715f5de) Successfully transitioned >> to start state for VM[User|giannitest] reservation id = >> 4b6c2f8d-a34c-4b86-a0d4-acd5ea87353a >> 2014-09-11 08:55:30,340 DEBUG [c.c.v.VirtualMachineManagerImpl] >> (Job-Executor-55:ctx-b5c34ec7 ctx-3715f5de) Trying to deploy VM, vm >> has dcId: 1 and podId: 1 >> 2014-09-11 08:55:30,340 DEBUG [c.c.v.VirtualMachineManagerImpl] >> (Job-Executor-55:ctx-b5c34ec7 ctx-3715f5de) Deploy avoids pods: [], >> clusters: [], hosts: [7] >> 2014-09-11 08:55:30,348 DEBUG [c.c.d.DeploymentPlanningManagerImpl] >> (Job-Executor-55:ctx-b5c34ec7 ctx-3715f5de) Deploy avoids pods: [], >> clusters: [], hosts: [7] >> 2014-09-11 08:55:30,348 DEBUG [c.c.d.DeploymentPlanningManagerImpl] > (Job-Executor-55:ctx-b5c34ec7 ctx-3715f5de) DataCenter id = '1' > provided is in avoid set, DeploymentPlanner cannot allocate the VM, returning. >> 2014-09-11 08:55:30,435 DEBUG [c.c.c.CapacityManagerImpl] >> (Job-Executor-55:ctx-b5c34ec7 ctx-3715f5de) VM state transitted from >> :Starting to Stopped with event: OperationFailedvm's original host id: >> null new host id: null host id before state transition: null >> 2014-09-11 08:55:30,477 DEBUG [c.c.v.UserVmManagerImpl] >> (Job-Executor-55:ctx-b5c34ec7 ctx-3715f5de) Destroying vm >> VM[User|giannitest] as it failed to create on Host with Id:null >> 2014-09-11 08:55:30,609 DEBUG [c.c.c.CapacityManagerImpl] >> (Job-Executor-55:ctx-b5c34ec7 ctx-3715f5de) VM state transitted from >> :Stopped to Error with event: OperationFailedToErrorvm's original >> host >> id: null new host id: null host id before state transition: null >> 2014-09-11 08:55:30,809 WARN [o.a.c.alerts] >> (Job-Executor-55:ctx-b5c34ec7 ctx-3715f5de) alertType:: 8 // >> dataCenterId:: 1 // podId:: 1 // clusterId:: null // message:: Failed >> to deploy Vm with Id: 86, on Host with Id: null >> 2014-09-11 08:55:30,932 DEBUG [c.c.a.m.AgentManagerImpl] >> (AgentManager-Handler-1:null) Ping from 6 >> 2014-09-11 08:55:31,078 ERROR [c.c.a.ApiAsyncJobDispatcher] >> (Job-Executor-55:ctx-b5c34ec7) Unexpected exception while executing >> org.apache.cloudstack.api.command.user.vm.DeployVMCmd >> com.cloud.utils.exception.CloudRuntimeException: Unable to start a VM >> due > to insufficient capacity >> at > com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java: > 605) >> at > org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.depl > oyVirt > ualMachine(VMEntityManagerImpl.java:237) >> at > org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl > .deplo > y(VirtualMachineEntityImpl.java:207) >> at > com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.j > ava:35 > 81) >> at > com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.j > ava:31 > 61) >> at > com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.j > ava:31 > 47) >> at sun.reflect.GeneratedMethodAccessor587.invoke(Unknown >> Source) ... >> >> >> Il 01/08/2014 11:44, Geoff Higginbottom ha scritto: >> Ugo, >> >> Login to the VR and tail the messages log then attempt to start the >> VM, > you should see various messages re DHCP updating etc in the VR. If > you don't see the VR update try restarting the network Iain the 'cleanup' > option, this will create a new VR. >> Regards >> >> Geoff Higginbottom >> CTO / Cloud Architect >> >> D: +44 20 3603 0542 | S: +44 20 3603 >> 0540 | M: +447968161581 >> >> geoff.higginbottom@shapeblue.com> c >> om> | >> www.shapeblue.com >> | Twitter:@cloudstackguru >> >> ShapeBlue Ltd, 53 Chandos Place, Covent Garden, London, WC2N >> 4HS >> >> >> On 1 Aug 2014, at 09:06, "Ugo Vasi" > it>> > wrote: >> Hi Geoff, >> I tried restarting r-29-VM but the problem is still blocking creation >> of > new VM in that isolated network. >> Creating a new VM in another isolated network, it works. Where can I >> look > at to find the problem? >> >> >> Il 31/07/2014 21:53, Geoff Higginbottom ha scritto: >> Ugo, >> >> Looks like CS is failing to update the VR with details for the new >> VM, try > restarting the VR. >> Regards >> >> Geoff Higginbottom >> CTO / Cloud Architect >> >> >> D: +44 20 3603 0542 | S: +44 20 3603 >> 0540 | M: +447968161581 >> >> geoff.higginbottom@shapeblue.com> c >> om>> om>m >> @shapeblue.com> | >> www.shapeblue.com >> < >> htp://www.shapeblue.com/> >> >> ShapeBlue Ltd, 53 Chandos Place, Covent Garden, London, WC2N >> 4HS >> >> >> >> On 31 Jul 2014, at 15:40, "Ugo Vasi" > it> ilto:ugo.vasi@procne.it>> wrote: >> Hi everybody, >> I installed a CloudStack 4.3.0 system with a manager (in a virtual > machine), two NFS primary storage, one NFS secondary storare and two > host agent on ubuntu distribution. >> The cluster have one advanced zone without security group >> configuration > with KVM hipervisors. >> I create some VM but now does not allow me to create more giving the >> error > message "Unable to start a VM due to insufficient capacity". >> I am sure that the resources (ip, ram, cpu and storage) are enough to > create new instances. In dashboard I see all ok. >> >From the management-server.log I see this errors messages (see >> >attachment > for full version) when I try to deploy a new virtual machine with > 1x1GHz cpu, 1GB or RAM and 20GB disk space: >> 2014-07-31 14:59:11,983 DEBUG [c.c.a.t.Request] >> (AgentManager-Handler-12:null) Seq 1-684594119: Processing: { Ans: , >> MgmtId: 77916390353436, via: 1, Ver: v1, Flags: 10, >> [{"com.cloud.agent.api.Answer":{"result":true,"wait":0}},{"com.cloud. >> a gent.api.Answer":{"result":false,"details":"VmDataCommand failed, >> check agent logs","wait":0}}] } >> 2014-07-31 14:59:11,983 INFO [c.c.v.VirtualMachineManagerImpl] > (Job-Executor-26:ctx-ab3aee38 ctx-d8600c16) Unable to contact resource. >> com.cloud.exception.ResourceUnavailableException: Resource >> [DataCenter:1] is unreachable: Unable to apply userdata and password >> entry on router >> 2014-07-31 14:59:12,453 DEBUG [c.c.c.CapacityManagerImpl] >> (Job-Executor-26:ctx-ab3aee38 ctx-d8600c16) VM state transitted from >> :Starting to Stopped with event: OperationFailedvm's original host id: >> null new host id: null host id before state transition: 7 >> 2014-07-31 14:59:12,677 DEBUG [c.c.c.CapacityManagerImpl] >> (Job-Executor-26:ctx-ab3aee38 ctx-d8600c16) VM state transitted from >> :Starting to Stopped with event: OperationFailedvm's original host id: >> null new host id: null host id before state transition: null >> 2014-07-31 14:59:12,708 DEBUG [c.c.v.UserVmManagerImpl] >> (Job-Executor-26:ctx-ab3aee38 ctx-d8600c16) Destroying vm >> VM[User|csweb2] as it failed to create on Host with Id:null >> 2014-07-31 14:59:12,738 DEBUG [c.c.c.CapacityManagerImpl] >> (Job-Executor-26:ctx-ab3aee38 ctx-d8600c16) VM state transitted from >> :Stopped to Error with event: OperationFailedToErrorvm's original >> host >> id: null new host id: null host id before state transition: null >> 2014-07-31 14:59:12,929 WARN [o.a.c.alerts] >> (Job-Executor-26:ctx-ab3aee38 ctx-d8600c16) alertType:: 8 // >> dataCenterId:: 1 // podId:: 1 // clusterId:: null // message:: Failed >> to deploy Vm with Id: 70, on Host with Id: null >> 2014-07-31 14:59:13,159 ERROR [c.c.a.ApiAsyncJobDispatcher] >> (Job-Executor-26:ctx-ab3aee38) Unexpected exception while executing >> org.apache.cloudstack.api.command.user.vm.DeployVMCmd >> com.cloud.utils.exception.CloudRuntimeException: Unable to start a VM >> due to insufficient capacity Caused by: >> com.cloud.exception.InsufficientServerCapacityException: Unable to >> create a deployment for VM[User|csweb2]Scope=interface >> com.cloud.dc.DataCenter; id=1 >> 2014-07-31 14:59:13,161 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] >> (Job-Executor-26:ctx-ab3aee38) Complete async job-783, jobStatus: >> FAILED, resultCode: 530, result: >> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList": >> [],"errorcode":530,"errortext":"Unable to start a VM due to >> insufficient capacity"} >> 2014-07-31 14:59:46,280 DEBUG [c.c.v.VirtualMachineManagerImpl] >> (Job-Executor-27:ctx-82be396e ctx-41f0cf24) Stopped called on >> VM[User|csweb2] but the state is Error >> 2014-07-31 14:59:46,330 DEBUG [c.c.c.CapacityManagerImpl] >> (Job-Executor-27:ctx-82be396e ctx-41f0cf24) VM state transitted from >> :Error to Expunging with event: DestroyRequestedvm's original host id: >> null new host id: null host id before state transition: null >> >> I tried to search in others log file ("VmDataCommand failed, check >> agent > logs"), but I don't found nothing... >> Can anyone Help me? >> >> -- >> >> U g o V a s i > it> ilto:ugo.vasi@procne.it>> >> P r o c n e s.r.l >) >> via Cotonificio 45 33010 Tavagnacco IT >> phone: +390432486523 fax: +390432486523 >> >> Le informazioni contenute in questo messaggio sono riservate e >> confidenziali ed ? vietata la diffusione in qualunque modo eseguita. >> Qualora Lei non fosse la persona a cui il presente messaggio ? >> destinato, La invitiamo ad eliminarlo e a non leggerlo, dandocene >> gentilmente comunicazione. >> Per qualsiasi informazione si prega di contattare > support@procne.it< > mailto > :support@procne.it> . >> Rif. D.L. 196/2003 >> >> >> >> >> >> Find out more about ShapeBlue and our range of CloudStack related >> services >> >> IaaS Cloud Design & >> Build >> CSForge - rapid IaaS deployment >> framework >> CloudStack Consulting >> CloudStack Infrastructure >> Support >> CloudStack Bootcamp Training >> Courses >> >> This email and any attachments to it may be confidential and are >> intended > solely for the use of the individual to whom it is addressed. Any > views or opinions expressed are solely those of the author and do not > necessarily represent those of Shape Blue Ltd or related companies. If > you are not the intended recipient of this email, you must neither > take any action based upon its contents, nor copy or show it to > anyone. Please contact the sender if you believe you have received > this email in error. Shape Blue Ltd is a company incorporated in > England & Wales. ShapeBlue Services India LLP is a company > incorporated in India and is operated under license from Shape Blue > Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in > Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA > Pty Ltd is a company registered by The Republic of South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is a registered trademark. >> >> >> >> >> >> -- >> >> U g o V a s i > it>> >> P r o c n e s.r.l >) >> via Cotonificio 45 33010 Tavagnacco IT >> phone: +390432486523 fax: +390432486523 >> >> Le informazioni contenute in questo messaggio sono riservate e >> confidenziali ed ? vietata la diffusione in qualunque modo eseguita. >> Qualora Lei non fosse la persona a cui il presente messaggio ? >> destinato, La invitiamo ad eliminarlo e a non leggerlo, dandocene >> gentilmente comunicazione. >> Per qualsiasi informazione si prega di contattare > support@procne.it . >> Rif. D.L. 196/2003 >> >> >> >> >> >> Find out more about ShapeBlue and our range of CloudStack related >> services >> >> IaaS Cloud Design & >> Build >> CSForge - rapid IaaS deployment >> framework >> CloudStack Consulting >> CloudStack Infrastructure >> Support >> CloudStack Bootcamp Training >> Courses >> >> This email and any attachments to it may be confidential and are >> intended > solely for the use of the individual to whom it is addressed. Any > views or opinions expressed are solely those of the author and do not > necessarily represent those of Shape Blue Ltd or related companies. If > you are not the intended recipient of this email, you must neither > take any action based upon its contents, nor copy or show it to > anyone. Please contact the sender if you believe you have received > this email in error. Shape Blue Ltd is a company incorporated in > England & Wales. ShapeBlue Services India LLP is a company > incorporated in India and is operated under license from Shape Blue > Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in > Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA > Pty Ltd is a company registered by The Republic of South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is a registered trademark. >> >> >> >> >> >> -- >> >> U g o V a s i > >> P r o c n e s.r.l >) >> via Cotonificio 45 33010 Tavagnacco IT >> phone: +390432486523 fax: +390432486523 >> >> Le informazioni contenute in questo messaggio sono riservate e >> confidenziali ed ? vietata la diffusione in qualunque modo eseguita. >> Qualora Lei non fosse la persona a cui il presente messaggio ? >> destinato, La invitiamo ad eliminarlo e a non leggerlo, dandocene >> gentilmente comunicazione. >> Per qualsiasi informazione si prega di contattare > support@procne.it . >> Rif. D.L. 196/2003 >> >> >> Find out more about ShapeBlue and our range of CloudStack related >> services >> >> IaaS Cloud Design & >> Build >> CSForge - rapid IaaS deployment >> framework >> CloudStack Consulting >> CloudStack Infrastructure >> Support >> CloudStack Bootcamp Training >> Courses >> >> This email and any attachments to it may be confidential and are >> intended > solely for the use of the individual to whom it is addressed. Any > views or opinions expressed are solely those of the author and do not > necessarily represent those of Shape Blue Ltd or related companies. If > you are not the intended recipient of this email, you must neither > take any action based upon its contents, nor copy or show it to > anyone. Please contact the sender if you believe you have received > this email in error. Shape Blue Ltd is a company incorporated in > England & Wales. ShapeBlue Services India LLP is a company > incorporated in India and is operated under license from Shape Blue > Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in > Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA > Pty Ltd is a company registered by The Republic of South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is a registered trademark. >> >> >> >