Return-Path: X-Original-To: apmail-cloudstack-issues-archive@www.apache.org Delivered-To: apmail-cloudstack-issues-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id C88FF105B0 for ; Fri, 9 Aug 2013 07:01:01 +0000 (UTC) Received: (qmail 50520 invoked by uid 500); 9 Aug 2013 07:01:00 -0000 Delivered-To: apmail-cloudstack-issues-archive@cloudstack.apache.org Received: (qmail 50500 invoked by uid 500); 9 Aug 2013 07:01:00 -0000 Mailing-List: contact issues-help@cloudstack.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cloudstack.apache.org Delivered-To: mailing list issues@cloudstack.apache.org Received: (qmail 50475 invoked by uid 500); 9 Aug 2013 07:00:59 -0000 Delivered-To: apmail-incubator-cloudstack-issues@incubator.apache.org Received: (qmail 50395 invoked by uid 99); 9 Aug 2013 07:00:57 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Aug 2013 07:00:57 +0000 Date: Fri, 9 Aug 2013 07:00:57 +0000 (UTC) From: "ASF subversion and git services (JIRA)" To: cloudstack-issues@incubator.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (CLOUDSTACK-2729) [Automation] Libvirt failed find primary storage and VM deployment failed MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/CLOUDSTACK-2729?page=3Dcom.atla= ssian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId= =3D13734490#comment-13734490 ]=20 ASF subversion and git services commented on CLOUDSTACK-2729: ------------------------------------------------------------- Commit 64bb7007053d004fe7a8ac3b610e20df69f16bab in branch refs/heads/4.1 fr= om [~weizhou] [ https://git-wip-us.apache.org/repos/asf?p=3Dcloudstack.git;h=3D64bb700 ] CLOUDSTACK-2729: increase waiting timeout for global file lock to 300 secon= ds (cherry picked from commit 6d86af85baa57bb20ec9642cbd1cd82c002dcfef) =20 > [Automation] Libvirt failed find primary storage and VM deployment faile= d =20 > -------------------------------------------------------------------------= --- > > Key: CLOUDSTACK-2729 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-272= 9 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the defa= ult.)=20 > Components: Storage Controller > Affects Versions: 4.2.0 > Environment: Master branch build > Automation environment - KVM =20 > Reporter: Rayees Namathponnan > Assignee: Wei Zhou > Priority: Blocker > Fix For: 4.2.0 > > Attachments: agent.log, cloud.dmp, CLOUDSTACK-2729.rar, managemen= t-server.rar > > > Failed to deploy VM in automation, environment created with , 2 hosts and= 2 primary storages in a cluster=20 > 1) /export/home/rayees/SC_QA_AUTO4/primary2 > 2) /export/home/rayees/SC_QA_AUTO4/primary > Libvirt failed to find the primary storage /export/home/rayees/SC_QA_AUTO= 4/primary and VM deployment failed=20 > MS log > 2013-05-28 21:11:44,540 DEBUG [agent.transport.Request] (consoleproxy-1:n= ull) Seq 4-936706756: Sending { Cmd , MgmtId: 29066118877352, via: 4, Ver:= v1, Flags: 100111, [{"StartCommand":{"vm":{"id":29,"name":"v-29-QA","type"= :"ConsoleProxy","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":1073741824,= "maxRam":1073741824,"arch":"x86_64","os":"Debian GNU/Linux 5.0 (32-bit)","b= ootArgs":" template=3DdomP type=3Dconsoleproxy host=3D10.223.49.195 port=3D= 8250 name=3Dv-29-QA premium=3Dtrue zone=3D1 pod=3D1 guid=3DProxy.29 proxy_v= m=3D29 disable_rp_filter=3Dtrue eth2ip=3D10.223.122.73 eth2mask=3D255.255.2= 55.192 gateway=3D10.223.122.65 eth0ip=3D169.254.0.154 eth0mask=3D255.255.0.= 0 eth1ip=3D10.223.50.96 eth1mask=3D255.255.255.192 mgmtcidr=3D10.223.49.192= /26 localgw=3D10.223.50.65 internaldns1=3D10.223.110.254 dns1=3D72.52.126.1= 1","rebootOnCrash":false,"enableHA":false,"limitCpuUse":false,"vncPassword"= :"d099568827911cef","params":{},"uuid":"5a146833-6a8c-44e5-83c0-50f34accf51= 3","disks":[{"id":32,"name":"ROOT-29","mountPoint":"/export/home/rayees/SC_= QA_AUTO4/primary","path":"f6f8d865-e9c0-4188-8a33-6c6383ca5075","size":2764= 06784,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid"= :"fff90cb5-06dd-33b3-8815-d78c08ca01d9","deviceId":0}],"nics":[{"deviceId":= 2,"networkRateMbps":-1,"defaultNic":true,"uuid":"21175978-96bd-4160-8228-8a= d15aa40c66","ip":"10.223.122.73","netmask":"255.255.255.192","gateway":"10.= 223.122.65","mac":"06:2e:5a:00:00:42","dns1":"72.52.126.11","broadcastType"= :"Vlan","type":"Public","broadcastUri":"vlan://1221","isolationUri":"vlan:/= /1221","isSecurityGroupEnabled":false},{"deviceId":0,"networkRateMbps":-1,"= defaultNic":false,"uuid":"e0651452-a76e-4564-96b2-3d5d51e9bcd6","ip":"169.2= 54.0.154","netmask":"255.255.0.0","gateway":"169.254.0.1","mac":"0e:00:a9:f= e:00:9a","broadcastType":"LinkLocal","type":"Control","isSecurityGroupEnabl= ed":false},{"deviceId":1,"networkRateMbps":-1,"defaultNic":false,"uuid":"07= 65c229-468e-4dfd-8382-24ac49791a8d","ip":"10.223.50.96","netmask":"255.255.= 255.192","gateway":"10.223.50.65","mac":"06:a8:e8:00:00:1d","broadcastType"= :"Native","type":"Management","isSecurityGroupEnabled":false}]},"hostIp":"1= 0.223.50.66","wait":0}},{"check.CheckSshCommand":{"ip":"169.254.0.154","por= t":3922,"interval":6,"retries":100,"name":"v-29-QA","wait":0}}] } > 2013-05-28 21:11:44,552 DEBUG [agent.transport.Request] (AgentManager-Han= dler-1:null) Seq 4-936706756: Processing: { Ans: , MgmtId: 29066118877352,= via: 4, Ver: v1, Flags: 110, [{"Answer":{"result":false,"details":"com.clo= ud.utils.exception.CloudRuntimeException: org.libvirt.LibvirtException: Sto= rage pool not found: no pool with matching uuid\n\tat com.cloud.hypervisor.= kvm.storage.LibvirtStorageAdaptor.getStoragePool(LibvirtStorageAdaptor.java= :380)\n\tat com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.getStora= gePool(KVMStoragePoolManager.java:72)\n\tat com.cloud.hypervisor.kvm.resour= ce.LibvirtComputingResource.createVbd(LibvirtComputingResource.java:3399)\n= \tat com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.execute(Lib= virtComputingResource.java:3293)\n\tat com.cloud.hypervisor.kvm.resource.Li= bvirtComputingResource.executeRequest(LibvirtComputingResource.java:1187)\n= \tat com.cloud.agent.Agent.processRequest(Agent.java:525)\n\tat com.cloud.a= gent.Agent$AgentRequestHandler.doTask(Agent.java:852)\n\tat com.cloud.utils= .nio.Task.run(Task.java:83)\n\tat java.util.concurrent.ThreadPoolExecutor.r= unWorker(ThreadPoolExecutor.java:1110)\n\tat java.util.concurrent.ThreadPoo= lExecutor$Worker.run(ThreadPoolExecutor.java:603)\n\tat java.lang.Thread.ru= n(Thread.java:679)\n","wait":0}},{"Answer":{"result":false,"details":"Stopp= ed by previous failure","wait":0}}] } > 2013-05-28 21:11:44,552 DEBUG [agent.manager.AgentAttache] (AgentManager-= Handler-1:null) Seq 4-936706756: No more commands found > 2013-05-28 21:11:44,552 DEBUG [agent.transport.Request] (consoleproxy-1:n= ull) Seq 4-936706756: Received: { Ans: , MgmtId: 29066118877352, via: 4, V= er: v1, Flags: 110, { Answer, Answer } } > 2013-05-28 21:11:44,559 ERROR [cloud.vm.VirtualMachineManagerImpl] (conso= leproxy-1:null) Failed to start instance VM[ConsoleProxy|v-29-QA] > com.cloud.utils.exception.CloudRuntimeException: Unable to get answer tha= t is of class com.cloud.agent.api.StartAnswer > at com.cloud.agent.manager.Commands.getAnswer(Commands.java:80) > at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMac= hineManagerImpl.java:873) > at com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineMan= agerImpl.java:550) > at com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineMan= agerImpl.java:543) > at com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(Cons= oleProxyManagerImpl.java:561) > at com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(C= onsoleProxyManagerImpl.java:923) > at com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(Cons= oleProxyManagerImpl.java:1667) > at com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(Cons= oleProxyManagerImpl.java:157) > at com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.= java:111) > Agent log > 2013-05-29 01:06:07,911 DEBUG [cloud.agent.Agent] (agentRequest-Handler-4= :null) Request:Seq 4-936706739: { Cmd , MgmtId: 29066118877352, via: 4, Ve= r: v1, Flags: 100111, [{"StartCommand":{"vm":{"id":29,"name":"v-29-QA","typ= e":"ConsoleProxy","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":107374182= 4,"maxRam":1073741824,"arch":"x86_64","os":"Debian GNU/Linux 5.0 (32-bit)",= "bootArgs":" template=3DdomP type=3Dconsoleproxy host=3D10.223.49.195 port= =3D8250 name=3Dv-29-QA premium=3Dtrue zone=3D1 pod=3D1 guid=3DProxy.29 prox= y_vm=3D29 disable_rp_filter=3Dtrue eth2ip=3D10.223.122.73 eth2mask=3D255.25= 5.255.192 gateway=3D10.223.122.65 eth0ip=3D169.254.1.243 eth0mask=3D255.255= .0.0 eth1ip=3D10.223.50.93 eth1mask=3D255.255.255.192 mgmtcidr=3D10.223.49.= 192/26 localgw=3D10.223.50.65 internaldns1=3D10.223.110.254 dns1=3D72.52.12= 6.11","rebootOnCrash":false,"enableHA":false,"limitCpuUse":false,"vncPasswo= rd":"d099568827911cef","params":{},"uuid":"5a146833-6a8c-44e5-83c0-50f34acc= f513","disks":[{"id":32,"name":"ROOT-29","mountPoint":"/export/home/rayees/= SC_QA_AUTO4/primary","path":"f6f8d865-e9c0-4188-8a33-6c6383ca5075","size":2= 76406784,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUu= id":"fff90cb5-06dd-33b3-8815-d78c08ca01d9","deviceId":0}],"nics":[{"deviceI= d":2,"networkRateMbps":-1,"defaultNic":true,"uuid":"21175978-96bd-4160-8228= -8ad15aa40c66","ip":"10.223.122.73","netmask":"255.255.255.192","gateway":"= 10.223.122.65","mac":"06:2e:5a:00:00:42","dns1":"72.52.126.11","broadcastTy= pe":"Vlan","type":"Public","broadcastUri":"vlan://1221","isolationUri":"vla= n://1221","isSecurityGroupEnabled":false},{"deviceId":0,"networkRateMbps":-= 1,"defaultNic":false,"uuid":"e0651452-a76e-4564-96b2-3d5d51e9bcd6","ip":"16= 9.254.1.243","netmask":"255.255.0.0","gateway":"169.254.0.1","mac":"0e:00:a= 9:fe:01:f3","broadcastType":"LinkLocal","type":"Control","isSecurityGroupEn= abled":false},{"deviceId":1,"networkRateMbps":-1,"defaultNic":false,"uuid":= "0765c229-468e-4dfd-8382-24ac49791a8d","ip":"10.223.50.93","netmask":"255.2= 55.255.192","gateway":"10.223.50.65","mac":"06:49:6a:00:00:1a","broadcastTy= pe":"Native","type":"Management","isSecurityGroupEnabled":false}]},"hostIp"= :"10.223.50.66","wait":0}},{"check.CheckSshCommand":{"ip":"169.254.1.243","= port":3922,"interval":6,"retries":100,"name":"v-29-QA","wait":0}}] } > 2013-05-29 01:06:07,912 DEBUG [cloud.agent.Agent] (agentRequest-Handler-4= :null) Processing command: com.cloud.agent.api.StartCommand > 2013-05-29 01:06:07,923 WARN [cloud.agent.Agent] (agentRequest-Handler-4= :null) Caught: > com.cloud.utils.exception.CloudRuntimeException: org.libvirt.LibvirtExcep= tion: Storage pool not found: no pool with matching uuid > at com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.getStor= agePool(LibvirtStorageAdaptor.java:380) > at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.getStor= agePool(KVMStoragePoolManager.java:72) > at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.cre= ateVbd(LibvirtComputingResource.java:3399) > at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.exe= cute(LibvirtComputingResource.java:3293) > at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.exe= cuteRequest(LibvirtComputingResource.java:1187) > at com.cloud.agent.Agent.processRequest(Agent.java:525) > at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:85= 2) > at com.cloud.utils.nio.Task.run(Task.java:83) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolEx= ecutor.java:1110) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolE= xecutor.java:603) > > create VM issuing create command with primary storage =E2=80=9C/export/= home/rayees/SC_QA_AUTO4/primary=E2=80=9D but its not available in pool-list > [root@Rack2Host11 ~]# virsh pool-list --all > Name State Autostart > ----------------------------------------- > 1c9fcb9a-aece-453b-8510-4de4b4dbffdc active no > 41b632b5-40b3-3024-a38b-ea259c72579f active no > [root@Rack2Host11 ~]# > [root@Rack2Host11 ~]# > [root@Rack2Host11 ~]# virsh pool-dumpxml 1c9fcb9a-aece-453b-8510-4de4b4db= ffdc > > 1c9fcb9a-aece-453b-8510-4de4b4dbffdc > 1c9fcb9a-aece-453b-8510-4de4b4dbffdc > 237777190912 > 2098745344 > 235678445568 > > > > /var/lib/libvirt/images > > 0700 > -1 > -1 > > > > [root@Rack2Host11 ~]# virsh pool-dumpxml 41b632b5-40b3-3024-a38b-ea259c72= 579f > > 41b632b5-40b3-3024-a38b-ea259c72579f > 41b632b5-40b3-3024-a38b-ea259c72579f > 11810778316800 > 9045745336320 > 2765032980480 > > > > > > > /mnt/41b632b5-40b3-3024-a38b-ea259c72579f > > 0700 > -1 > -1 > > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrato= rs For more information on JIRA, see: http://www.atlassian.com/software/jira