cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Edison Su <Edison...@citrix.com>
Subject RE: [jira] [Commented] (CLOUDSTACK-2729) [Automation] Libvirt failed find primary storage and VM deployment failed
Date Thu, 08 Aug 2013 00:15:00 GMT
The local path of storage pool should be able to get from LibvirtStoragePoolDef.

> -----Original Message-----
> From: Marcus Sorensen [mailto:shadowsor@gmail.com]
> Sent: Wednesday, August 07, 2013 4:43 PM
> To: dev@cloudstack.apache.org
> Cc: cloudstack-issues@incubator.apache.org
> Subject: Re: [jira] [Commented] (CLOUDSTACK-2729) [Automation] Libvirt
> failed find primary storage and VM deployment failed
> 
> Here's the debugging I put in:
> 
> 2013-08-07 17:39:12,286 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-3:null) attempting to lock pool
> 2a9c418e-dd56-3f2c-9081-032b877d5644 with file '/mnt/2a9c418e-dd56-3f2c-
> 9081-032b877d5644/KVMFILELOCK/.lock'
> 2013-08-07 17:39:12,425 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-3:null) attempting to lock pool
> 2fa10069-1fb3-4eea-8dc4-f773e7be0d19 with file '/mnt/2fa10069-1fb3-4eea-
> 8dc4-f773e7be0d19/KVMFILELOCK/.lock'
> 2013-08-07 17:39:12,479 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-3:null) Seq 1-1276706825:  { Ans: , MgmtId:
> 90520735907663, via: 1, Ver: v1, Flags: 110,
> [{"storage.PrimaryStorageDownloadAnswer":{"templateSize":0,"result":fals
> e,"details":"com.cloud.utils.exception.CloudRuntimeException:
> Can not get file lock to refresh the pool 2fa10069-1fb3-4eea-8dc4-
> f773e7be0d19","wait":0}}] }
> 2013-08-07 17:39:12,536 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-4:null) Request:Seq 1-1276706826:  { Cmd ,
> MgmtId: 90520735907663, via: 1, Ver: v1, Flags: 100111,
> [{"StopCommand":{"isProxy":false,"vmName":"s-57-VM","wait":0}}] }
> 
> Note that 2a9c418e-dd56-3f2c-9081-032b877d5644 is my secondary storage,
> 2fa10069-1fb3-4eea-8dc4-f773e7be0d19 is local storage in
> /var/lib/libvirt/images
> 
> On Wed, Aug 7, 2013 at 5:41 PM, Marcus Sorensen <shadowsor@gmail.com>
> wrote:
> > Here's the issue... local storage isn't mounted like remote storage is.
> >
> > This is invalid for local storage:
> >
> > String lockFile = _mountPoint + File.separator + pool.getUUIDString()
> > + File.separator + _lockfile;
> >
> > Need to look at storage pool type and change lockfile to path for
> > local storage
> >
> > On Wed, Aug 7, 2013 at 5:18 PM, Marcus Sorensen (JIRA) <jira@apache.org>
> wrote:
> >>
> >>     [
> >> https://issues.apache.org/jira/browse/CLOUDSTACK-
> 2729?page=com.atlass
> >> ian.jira.plugin.system.issuetabpanels:comment-
> tabpanel&focusedComment
> >> Id=13732906#comment-13732906 ]
> >>
> >> Marcus Sorensen commented on CLOUDSTACK-2729:
> >> ---------------------------------------------
> >>
> >> Sorry, I think the KVMFILELOCK is left over on secondary storage
> >> because it's unable to lock the primary storage, and doesn't clean up
> >> after itself. I'm not sure why it can't take the lock out on primary
> >> yet... troubleshooting that.
> >>
> >>
> >>
> >>> [Automation] Libvirt failed find primary  storage and VM deployment
> >>> failed
> >>> --------------------------------------------------------------------
> >>> --------
> >>>
> >>>                 Key: CLOUDSTACK-2729
> >>>                 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-2729
> >>>             Project: CloudStack
> >>>          Issue Type: Bug
> >>>      Security Level: Public(Anyone can view this level - this is the default.)
> >>>          Components: Storage Controller
> >>>    Affects Versions: 4.2.0
> >>>         Environment: Master branch build Automation environment -
> >>> KVM
> >>>            Reporter: Rayees Namathponnan
> >>>            Assignee: edison su
> >>>            Priority: Blocker
> >>>             Fix For: 4.2.0
> >>>
> >>>         Attachments: CLOUDSTACK-2729.rar
> >>>
> >>>
> >>> Failed to deploy VM in automation, environment created with , 2
> >>> hosts and 2 primary storages in a cluster
> >>> 1) /export/home/rayees/SC_QA_AUTO4/primary2
> >>> 2) /export/home/rayees/SC_QA_AUTO4/primary
> >>> Libvirt failed to find the primary storage
> >>> /export/home/rayees/SC_QA_AUTO4/primary and VM deployment
> failed MS
> >>> log
> >>> 2013-05-28 21:11:44,540 DEBUG [agent.transport.Request]
> >>> (consoleproxy-1:null) Seq 4-936706756: Sending  { Cmd , MgmtId:
> >>> 29066118877352, via: 4, Ver: v1, Flags: 100111,
> >>> [{"StartCommand":{"vm":{"id":29,"name":"v-29-
> QA","type":"ConsoleProx
> >>>
> y","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":1073741824,"maxRa
> >>> m":1073741824,"arch":"x86_64","os":"Debian GNU/Linux 5.0
> >>> (32-bit)","bootArgs":" template=domP type=consoleproxy
> >>> host=10.223.49.195 port=8250 name=v-29-QA premium=true zone=1
> pod=1
> >>> guid=Proxy.29 proxy_vm=29 disable_rp_filter=true
> >>> eth2ip=10.223.122.73 eth2mask=255.255.255.192 gateway=10.223.122.65
> >>> eth0ip=169.254.0.154 eth0mask=255.255.0.0 eth1ip=10.223.50.96
> >>> eth1mask=255.255.255.192 mgmtcidr=10.223.49.192/26
> >>> localgw=10.223.50.65 internaldns1=10.223.110.254
> >>> dns1=72.52.126.11","rebootOnCrash":false,"enableHA":false,"limitCpuU
> >>> se":false,"vncPassword":"d099568827911cef","params":{},"uuid":"5a146
> >>> 833-6a8c-44e5-83c0-50f34accf513","disks":[{"id":32,"name":"ROOT-29",
> >>>
> "mountPoint":"/export/home/rayees/SC_QA_AUTO4/primary","path":"f6f8
> d
> >>> 865-e9c0-4188-8a33-6c6383ca5075","size":276406784,"type":"ROOT","sto
> >>> ragePoolType":"NetworkFilesystem","storagePoolUuid":"fff90cb5-06dd-
> 3
> >>> 3b3-8815-d78c08ca01d9","deviceId":0}],"nics":[{"deviceId":2,"network
> >>> RateMbps":-1,"defaultNic":true,"uuid":"21175978-96bd-4160-8228-
> 8ad15
> >>> aa40c66","ip":"10.223.122.73","netmask":"255.255.255.192","gateway":
> >>> "10.223.122.65","mac":"06:2e:5a:00:00:42","dns1":"72.52.126.11","bro
> >>> adcastType":"Vlan","type":"Public","broadcastUri":"vlan://1221","iso
> >>> lationUri":"vlan://1221","isSecurityGroupEnabled":false},{"deviceId"
> >>> :0,"networkRateMbps":-1,"defaultNic":false,"uuid":"e0651452-a76e-456
> >>> 4-96b2-3d5d51e9bcd6","ip":"169.254.0.154","netmask":"255.255.0.0","g
> >>> ateway":"169.254.0.1","mac":"0e:00:a9:fe:00:9a","broadcastType":"Lin
> >>> kLocal","type":"Control","isSecurityGroupEnabled":false},{"deviceId"
> >>> :1,"networkRateMbps":-1,"defaultNic":false,"uuid":"0765c229-468e-4df
> >>> d-8382-24ac49791a8d","ip":"10.223.50.96","netmask":"255.255.255.192"
> >>> ,"gateway":"10.223.50.65","mac":"06:a8:e8:00:00:1d","broadcastType":
> >>> "Native","type":"Management","isSecurityGroupEnabled":false}]},"host
> >>> Ip":"10.223.50.66","wait":0}},{"check.CheckSshCommand":{"ip":"169.25
> >>> 4.0.154","port":3922,"interval":6,"retries":100,"name":"v-29-QA","wa
> >>> it":0}}] }
> >>> 2013-05-28 21:11:44,552 DEBUG [agent.transport.Request]
> >>> (AgentManager-Handler-1:null) Seq 4-936706756: Processing:  { Ans: ,
> >>> MgmtId: 29066118877352, via: 4, Ver: v1, Flags: 110,
> >>> [{"Answer":{"result":false,"details":"com.cloud.utils.exception.Clou
> >>> dRuntimeException: org.libvirt.LibvirtException: Storage pool not
> >>> found: no pool with matching uuid\n\tat
> >>>
> com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.getStoragePoo
> >>> l(LibvirtStorageAdaptor.java:380)\n\tat
> >>>
> com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.getStoragePoo
> >>> l(KVMStoragePoolManager.java:72)\n\tat
> >>>
> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.createVbd
> >>> (LibvirtComputingResource.java:3399)\n\tat
> >>>
> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.execute(L
> >>> ibvirtComputingResource.java:3293)\n\tat
> >>>
> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRe
> >>> quest(LibvirtComputingResource.java:1187)\n\tat
> >>> com.cloud.agent.Agent.processRequest(Agent.java:525)\n\tat
> >>>
> com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:852)\n\t
> >>> at com.cloud.utils.nio.Task.run(Task.java:83)\n\tat
> >>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor
> >>> .java:1110)\n\tat
> >>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecuto
> >>> r.java:603)\n\tat
> >>> java.lang.Thread.run(Thread.java:679)\n","wait":0}},{"Answer":{"resu
> >>> lt":false,"details":"Stopped by previous failure","wait":0}}] }
> >>> 2013-05-28 21:11:44,552 DEBUG [agent.manager.AgentAttache]
> >>> (AgentManager-Handler-1:null) Seq 4-936706756: No more commands
> >>> found
> >>> 2013-05-28 21:11:44,552 DEBUG [agent.transport.Request]
> >>> (consoleproxy-1:null) Seq 4-936706756: Received:  { Ans: , MgmtId:
> >>> 29066118877352, via: 4, Ver: v1, Flags: 110, { Answer, Answer } }
> >>> 2013-05-28 21:11:44,559 ERROR [cloud.vm.VirtualMachineManagerImpl]
> >>> (consoleproxy-1:null) Failed to start instance
> >>> VM[ConsoleProxy|v-29-QA]
> >>> com.cloud.utils.exception.CloudRuntimeException: Unable to get
> answer that is of class com.cloud.agent.api.StartAnswer
> >>>         at
> com.cloud.agent.manager.Commands.getAnswer(Commands.java:80)
> >>>         at
> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineMa
> nagerImpl.java:873)
> >>>         at
> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerIm
> pl.java:550)
> >>>         at
> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerIm
> pl.java:543)
> >>>         at
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(ConsoleProx
> yManagerImpl.java:561)
> >>>         at
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsolePr
> oxyManagerImpl.java:923)
> >>>         at
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsolePr
> oxyManagerImpl.java:1667)
> >>>         at
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsolePr
> oxyManagerImpl.java:157)
> >>>         at
> >>>
> com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java
> :1
> >>> 11)
> >>> Agent log
> >>> 2013-05-29 01:06:07,911 DEBUG [cloud.agent.Agent]
> >>> (agentRequest-Handler-4:null) Request:Seq 4-936706739:  { Cmd ,
> >>> MgmtId: 29066118877352, via: 4, Ver: v1, Flags: 100111,
> >>> [{"StartCommand":{"vm":{"id":29,"name":"v-29-
> QA","type":"ConsoleProx
> >>>
> y","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":1073741824,"maxRa
> >>> m":1073741824,"arch":"x86_64","os":"Debian GNU/Linux 5.0
> >>> (32-bit)","bootArgs":" template=domP type=consoleproxy
> >>> host=10.223.49.195 port=8250 name=v-29-QA premium=true zone=1
> pod=1
> >>> guid=Proxy.29 proxy_vm=29 disable_rp_filter=true
> >>> eth2ip=10.223.122.73 eth2mask=255.255.255.192 gateway=10.223.122.65
> >>> eth0ip=169.254.1.243 eth0mask=255.255.0.0 eth1ip=10.223.50.93
> >>> eth1mask=255.255.255.192 mgmtcidr=10.223.49.192/26
> >>> localgw=10.223.50.65 internaldns1=10.223.110.254
> >>> dns1=72.52.126.11","rebootOnCrash":false,"enableHA":false,"limitCpuU
> >>> se":false,"vncPassword":"d099568827911cef","params":{},"uuid":"5a146
> >>> 833-6a8c-44e5-83c0-50f34accf513","disks":[{"id":32,"name":"ROOT-29",
> >>>
> "mountPoint":"/export/home/rayees/SC_QA_AUTO4/primary","path":"f6f8
> d
> >>> 865-e9c0-4188-8a33-6c6383ca5075","size":276406784,"type":"ROOT","sto
> >>> ragePoolType":"NetworkFilesystem","storagePoolUuid":"fff90cb5-06dd-
> 3
> >>> 3b3-8815-d78c08ca01d9","deviceId":0}],"nics":[{"deviceId":2,"network
> >>> RateMbps":-1,"defaultNic":true,"uuid":"21175978-96bd-4160-8228-
> 8ad15
> >>> aa40c66","ip":"10.223.122.73","netmask":"255.255.255.192","gateway":
> >>> "10.223.122.65","mac":"06:2e:5a:00:00:42","dns1":"72.52.126.11","bro
> >>> adcastType":"Vlan","type":"Public","broadcastUri":"vlan://1221","iso
> >>> lationUri":"vlan://1221","isSecurityGroupEnabled":false},{"deviceId"
> >>> :0,"networkRateMbps":-1,"defaultNic":false,"uuid":"e0651452-a76e-456
> >>> 4-96b2-3d5d51e9bcd6","ip":"169.254.1.243","netmask":"255.255.0.0","g
> >>> ateway":"169.254.0.1","mac":"0e:00:a9:fe:01:f3","broadcastType":"Lin
> >>> kLocal","type":"Control","isSecurityGroupEnabled":false},{"deviceId"
> >>> :1,"networkRateMbps":-1,"defaultNic":false,"uuid":"0765c229-468e-4df
> >>> d-8382-24ac49791a8d","ip":"10.223.50.93","netmask":"255.255.255.192"
> >>> ,"gateway":"10.223.50.65","mac":"06:49:6a:00:00:1a","broadcastType":
> >>> "Native","type":"Management","isSecurityGroupEnabled":false}]},"host
> >>> Ip":"10.223.50.66","wait":0}},{"check.CheckSshCommand":{"ip":"169.25
> >>> 4.1.243","port":3922,"interval":6,"retries":100,"name":"v-29-QA","wa
> >>> it":0}}] }
> >>> 2013-05-29 01:06:07,912 DEBUG [cloud.agent.Agent]
> >>> (agentRequest-Handler-4:null) Processing command:
> >>> com.cloud.agent.api.StartCommand
> >>> 2013-05-29 01:06:07,923 WARN  [cloud.agent.Agent] (agentRequest-
> Handler-4:null) Caught:
> >>> com.cloud.utils.exception.CloudRuntimeException:
> org.libvirt.LibvirtException: Storage pool not found: no pool with matching
> uuid
> >>>         at
> com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.getStoragePool(Lib
> virtStorageAdaptor.java:380)
> >>>         at
> com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.getStoragePoo
> l(KVMStoragePoolManager.java:72)
> >>>         at
> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.createVbd(Li
> bvirtComputingResource.java:3399)
> >>>         at
> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.execute(Libv
> irtComputingResource.java:3293)
> >>>         at
> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeReq
> uest(LibvirtComputingResource.java:1187)
> >>>         at com.cloud.agent.Agent.processRequest(Agent.java:525)
> >>>         at
> com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:852)
> >>>         at com.cloud.utils.nio.Task.run(Task.java:83)
> >>>         at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
> a:1110)
> >>>         at
> >>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecuto
> >>> r.java:603)
> >>> > create VM issuing create command with primary storage
> >>> > "/export/home/rayees/SC_QA_AUTO4/primary" but its not available
> in
> >>> > pool-list
> >>> [root@Rack2Host11 ~]# virsh pool-list --all
> >>> Name                 State      Autostart
> >>> -----------------------------------------
> >>> 1c9fcb9a-aece-453b-8510-4de4b4dbffdc active     no
> >>> 41b632b5-40b3-3024-a38b-ea259c72579f active     no
> >>> [root@Rack2Host11 ~]#
> >>> [root@Rack2Host11 ~]#
> >>> [root@Rack2Host11 ~]# virsh pool-dumpxml
> >>> 1c9fcb9a-aece-453b-8510-4de4b4dbffdc
> >>> <pool type='dir'>
> >>>   <name>1c9fcb9a-aece-453b-8510-4de4b4dbffdc</name>
> >>>   <uuid>1c9fcb9a-aece-453b-8510-4de4b4dbffdc</uuid>
> >>>   <capacity unit='bytes'>237777190912</capacity>
> >>>   <allocation unit='bytes'>2098745344</allocation>
> >>>   <available unit='bytes'>235678445568</available>
> >>>   <source>
> >>>   </source>
> >>>   <target>
> >>>     <path>/var/lib/libvirt/images</path>
> >>>     <permissions>
> >>>       <mode>0700</mode>
> >>>       <owner>-1</owner>
> >>>       <group>-1</group>
> >>>     </permissions>
> >>>   </target>
> >>> </pool>
> >>> [root@Rack2Host11 ~]# virsh pool-dumpxml
> >>> 41b632b5-40b3-3024-a38b-ea259c72579f
> >>> <pool type='netfs'>
> >>>   <name>41b632b5-40b3-3024-a38b-ea259c72579f</name>
> >>>   <uuid>41b632b5-40b3-3024-a38b-ea259c72579f</uuid>
> >>>   <capacity unit='bytes'>11810778316800</capacity>
> >>>   <allocation unit='bytes'>9045745336320</allocation>
> >>>   <available unit='bytes'>2765032980480</available>
> >>>   <source>
> >>>     <host name='10.223.110.232'/>
> >>>     <dir path='/export/home/rayees/SC_QA_AUTO4/primary2'/>
> >>>     <format type='auto'/>
> >>>   </source>
> >>>   <target>
> >>>     <path>/mnt/41b632b5-40b3-3024-a38b-ea259c72579f</path>
> >>>     <permissions>
> >>>       <mode>0700</mode>
> >>>       <owner>-1</owner>
> >>>       <group>-1</group>
> >>>     </permissions>
> >>>   </target>
> >>> </pool>
> >>
> >> --
> >> This message is automatically generated by JIRA.
> >> If you think it was sent incorrectly, please contact your JIRA
> >> administrators For more information on JIRA, see:
> http://www.atlassian.com/software/jira

Mime
View raw message