incubator-cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Serg <kernc...@gmail.com>
Subject Re: 3.0.2 - KVM Agent problem? Please help.
Date Thu, 07 Jun 2012 18:33:14 GMT
Hi,

You right, it looking like bug in libvirt 0.9.4-23.el6_2.4 
I get error from libvirt  when call the "pool-create"

Workaround before upgrade libvirt, is unmount mounted "sec. storage" and re-create it again.
Then, libvirt can create the new pool's 

--
Serg


On Jun 6, 2012, at 1:14 AM, Edison Su wrote:

> Sounds like your libvirt can’t create a storage pool. Can you try the following command:
> 1.        
> cat >> storage.xml << EOF
> <pool type='netfs'>
> <name>304424d9-db34-40d9-b0a7-3bd689b0b862</name>
> <uuid>304424d9-db34-40d9-b0a7-3bd689b0b862</uuid>
> <source>
> <host name='NFS.DOMAIN.COM'/>
> <dir path='/opt/secondary'/>
> </source>
> <target>
> <path>/mnt/304424d9-db34-40d9-b0a7-3bd689b0b862</path>
> </target>
> </pool>
>  
> 2.       virsh pool-create storage.xml
> 3.       Any error in the above command? If you get the error like “cannot open volume
'/mnt/304424d9-db34-40d9-b0a7-3bd689b0b862/template”, can you upgrade libvirt to
> Installed Packages
> Name        : libvirt
> Arch        : x86_64
> Version     : 0.9.4
> Release     : 23.el6_2.8
> Size        : 4.0 M
>  
> From: Serg [mailto:kerncore@gmail.com] 
> Sent: Tuesday, June 05, 2012 11:48 AM
> To: Edison Su
> Cc: cloudstack-dev@incubator.apache.org
> Subject: Re: 3.0.2 - KVM Agent problem? Please help.
>  
> Hi , 
>  
> Thanks for your patch. It's solved the critical problem with Re-Start the instances !
=)
> ( In my setup only HA instances have shared storage )
>  
> But problem with snapshot's on "Running Instances" still exist 
>  
> 2012-06-05 19:15:52,853 DEBUG [cloud.agent.Agent] (agentRequest-Handler-2:null) Request:Seq
34-77010656:  { Cmd , MgmtId: 207382744522, via: 34, Ver: v1, Flags: 100011, [{"BackupSnapshotCommand":{"prevSnapshotUuid":"/var/lib/libvirt/images/4341f531-bef6-47b6-ae75-2b8d6de63038/i-8-403-VM_ROOT-403_20120605164038","prevBackupUuid":"/snapshots/5/8/442/i-8-403-VM_ROOT-403_20120605164038","isVolumeInactive":false,"vmName":"i-8-403-VM","snapshotId":1746,"pool":{"id":208,"uuid":"c6f1f032-5aad-4983-bc83-0ec1a6b5e32c","host":"111.111.111.111","path":"/var/lib/libvirt/images/","port":0,"type":"Filesystem"},"primaryStoragePoolNameLabel":"c6f1f032-5aad-4983-bc83-0ec1a6b5e32c","snapshotUuid":"/var/lib/libvirt/images/4341f531-bef6-47b6-ae75-2b8d6de63038/i-8-403-VM_ROOT-403_20120605170902","snapshotName":"i-8-403-VM_ROOT-403_20120605170902","secondaryStorageUrl":"nfs://NFS-URL/opt/secondary","dcId":5,"accountId":8,"volumeId":442,"volumePath":"4341f531-bef6-47b6-ae75-2b8d6de63038","wait":10800}}]
}
>  
> Then "Job" create new directory under /mnt/ for access to template ???
>  
> 2012-06-05 19:15:53,558 DEBUG [agent.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:null)org.libvirt.LibvirtException:
cannot open volume '/mnt/304424d9-db34-40d9-b0a7-3bd689b0b862/template':Is a directory
>  
>  
> Full log of job : 
> http://pastebin.com/fE1M1BfS
>  
> UI  : Error  Popup
> " Created snapshot: com.cloud.storage.SnapshotVO$$EnhancerByCGLIB$$4d81b9c7@78ab83a on
primary but failed to backup on secondary"
>  
> Can I provide some additional logs ?
>  
>  
>  
> --
> Serg
>  
>  
>  
> On Jun 5, 2012, at 12:20 AM, Edison Su wrote:
> 
> 
> It's a bug in kvm agent code, I fixed the upgrade issue from 2.2.x to 3.0.x, but it only
worked for shared storage. 
> You can fix it by update db with the following sql:
> 
> update volumes set path=SUBSTRING_INDEX(`path`, '/', -1);
> 
> , then restart mgt server.
> 
> 
> -----Original Message-----
> From: Serg [mailto:kerncore@gmail.com]
> Sent: Monday, June 04, 2012 1:25 PM
> To: Edison Su
> Cc: cloudstack-dev@incubator.apache.org
> Subject: Re: 3.0.2 - KVM Agent problem? Please help.
>  
> Hi,
>  
> Thanks for your response,
>  
> I turned on debug logging but nothing new has been discovered.
> after upgrading from 2.2.14 to 3.0.2 all KVM server with Local Storage
> behave identically
>  
> - Unable to re-start the instance ( instance can't be started anymore )
> - Make a snapshot for volume.
> - Can't download the volume
>  
> For now, back to online such instances I'm can by the following way
> - import volume of stopped instance as template and the start new
> instance from
>  
> Log :
> libvir: Storage error : Storage volume not found: no storage vol with
> matching name
>  
> [root@ /]# virsh vol-list c6f1f032-5aad-4983-bc83-0ec1a6b5e32c | grep
> 4341f531-bef6-47b6-ae75-2b8d6de63038
> 4341f531-bef6-47b6-ae75-2b8d6de63038 /var/lib/libvirt/images/4341f531-
> bef6-47b6-ae75-2b8d6de63038
>  
> [root@ /]# qemu-img check /var/lib/libvirt/images/4341f531-bef6-47b6-
> ae75-2b8d6de63038
> No errors were found on the image.
>  
> Full log output :
>  
> Log output : http://pastebin.com/dd0Wz22a
>  
>  
> --
> Serg
>  
>  
>  
>  
> On Jun 4, 2012, at 8:33 PM, Edison Su wrote:
>  
> Please turn log level to debug(/etc/cloud/agent/log4j-cloud.xml) on
> your kvm host, then restart cloud-agent, take a look at agent log again.
>  
> -----Original Message-----
> From: Serg [mailto:kerncore@gmail.com]
> Sent: Sunday, June 03, 2012 11:48 AM
> To: cloudstack-dev@incubator.apache.org
> Subject: 3.0.2 - KVM Agent problem? Please help.
>  
> Hi,
>  
>  
> How to debug such problem ?
> All volumes exist and instances running, but can't to
>  
> CentOS release 6.2 (Final)
> libvirt-0.9.4-23.el6_2.4.x86_64
> libvirt-client-0.9.4-23.el6_2.4.x86_64
>  
>  
> [root@kvmhost2 snapshot]# tail -10 /var/log/cloud/agent/agent.log
> com.cloud.utils.exception.CloudRuntimeException:
> org.libvirt.LibvirtException: Storage volume not found: no storage
> vol
> with matching name '/var/lib/libvirt/images/67033fb9-a432-4162-8b22-
> d8a7b2aeb100'
>           at
>  
> com.cloud.agent.storage.LibvirtStorageAdaptor.getVolume(LibvirtStorageA
> daptor.java:81)
> ...cut...
> libvir: Storage error : Storage volume not found: no storage vol
> with
> matching name '/opt/primary/f5b97a37-f8dd-485c-945c-da31d20b9e23'
> ...cut...
> libvir: Storage error : Storage volume not found: no storage vol
> with
> matching name '/var/lib/libvirt/images/58ba9eed-c864-44f3-a987-
> 9d1eaab182e1'
>  
> [root@kvmhost2 snapshot]# tail -10 /var/log/cloud/agent/agent.log |
> awk
> '{print $15}' | xargs file
> /opt/primary/e371ebb7-8d9e-4c64-83e9-effea15a00e7:            Qemu
> Image, Format: Qcow , Version: 2
> /opt/primary/e371ebb7-8d9e-4c64-83e9-effea15a00e7:            Qemu
> Image, Format: Qcow , Version: 2
> /opt/primary/e55358fa-8dc5-4070-b3b6-3f4bc01b17ae:            Qemu
> Image, Format: Qcow , Version: 2
> /opt/primary/e55358fa-8dc5-4070-b3b6-3f4bc01b17ae:            Qemu
> Image, Format: Qcow , Version: 2
> /var/lib/libvirt/images/d49b318c-8bb4-4bb2-a897-1c40ef234d30: Qemu
> Image, Format: Qcow , Version: 2
> /var/lib/libvirt/images/d49b318c-8bb4-4bb2-a897-1c40ef234d30: Qemu
> Image, Format: Qcow , Version: 2
> /var/lib/libvirt/images/193c378c-80ba-4b82-affe-45c9da0ee839: Qemu
> Image, Format: Qcow , Version: 2
> /var/lib/libvirt/images/193c378c-80ba-4b82-affe-45c9da0ee839: Qemu
> Image, Format: Qcow , Version: 2
> /var/lib/libvirt/images/58ba9eed-c864-44f3-a987-9d1eaab182e1: Qemu
> Image, Format: Qcow , Version: 2
> /var/lib/libvirt/images/58ba9eed-c864-44f3-a987-9d1eaab182e1: Qemu
> Image, Format: Qcow , Version: 2
>  
> When I'm try to start instance through UI I'm get the following
> error :
> UI Error pop-up : Resource [Host:34] is unreachable: Host 34: Unable
> to
> start instance due to Unable to get answer that is of class
> com.cloud.agent.api.StartAnswer
>  
> Agent LOG :
>  
> 2012-06-03 20:47:13,397 WARN
> [resource.computing.LibvirtComputingResource] (agentRequest-Handler-
> 2:null) Exception
> com.cloud.utils.exception.CloudRuntimeException:
> org.libvirt.LibvirtException: Storage volume not found: no storage
> vol
> with matching name '/var/lib/libvirt/images/19d086dd-8229-48c2-aaa7-
> ea3d218dc5dc'
>           at
>  
> com.cloud.agent.storage.LibvirtStorageAdaptor.getVolume(LibvirtStorageA
> daptor.java:81)
>           at
>  
> com.cloud.agent.storage.LibvirtStorageAdaptor.getPhysicalDisk(LibvirtSt
> orageAdaptor.java:450)
>           at
>  
> com.cloud.agent.storage.LibvirtStoragePool.getPhysicalDisk(LibvirtStora
> gePool.java:104)
>           at
>  
> com.cloud.agent.resource.computing.LibvirtComputingResource.createVbd(L
> ibvirtComputingResource.java:2593)
>           at
>  
> com.cloud.agent.resource.computing.LibvirtComputingResource.execute(Lib
> virtComputingResource.java:2510)
>           at
>  
> com.cloud.agent.resource.computing.LibvirtComputingResource.executeRequ
> est(LibvirtComputingResource.java:985)
>           at com.cloud.agent.Agent.processRequest(Agent.java:517)
>           at
> com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:827)
>           at com.cloud.utils.nio.Task.run(Task.java:79)
>           at
>  
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.ja
> va:1110)
>           at
>  
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.j
> ava:603)
>           at java.lang.Thread.run(Thread.java:679)
> 2012-06-03 20:47:13,399{GMT} WARN  [cloud.agent.Agent]
> (agentRequest-
> Handler-2:) Caught:
> java.lang.NullPointerException
>           at
>  
> com.cloud.agent.resource.computing.LibvirtComputingResource.cleanupVMNe
> tworks(LibvirtComputingResource.java:3664)
>           at
>  
> com.cloud.agent.resource.computing.LibvirtComputingResource.handleVmSta
> rtFailure(LibvirtComputingResource.java:2416)
>           at
>  
> com.cloud.agent.resource.computing.LibvirtComputingResource.execute(Lib
> virtComputingResource.java:2539)
>           at
>  
> com.cloud.agent.resource.computing.LibvirtComputingResource.executeRequ
> est(LibvirtComputingResource.java:985)
>           at com.cloud.agent.Agent.processRequest(Agent.java:517)
>           at
> com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:827)
>           at com.cloud.utils.nio.Task.run(Task.java:79)
>           at
>  
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.ja
> va:1110)
>           at
>  
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.j
> ava:603)
>           at java.lang.Thread.run(Thread.java:679)
>  
>  
>  
> --
> Serg
>  
>  


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message