incubator-cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Edison Su <Edison...@citrix.com>
Subject RE: 3.0.2 - KVM Agent problem? Please help.
Date Thu, 07 Jun 2012 18:14:49 GMT


> -----Original Message-----
> From: de Jong, Mark-Jan [mailto:Mark-Jan.deJong@teoco.com]
> Sent: Thursday, June 07, 2012 11:04 AM
> To: Edison Su; 'cloudstack-dev@incubator.apache.org'
> Subject: RE: 3.0.2 - KVM Agent problem? Please help.
> 
> 
> 
> > -----Original Message-----
> > From: Edison Su [mailto:Edison.su@citrix.com]
> > Sent: Thursday, June 07, 2012 1:54 PM
> > To: 'cloudstack-dev@incubator.apache.org'; de Jong, Mark-Jan
> > Subject: RE: 3.0.2 - KVM Agent problem? Please help.
> >
> > Thanks for your report, I'll fix it in the future release(not in
> 3.0.3
> > though).
> > And a bug is created(http://bugs.cloudstack.org/browse/CS-15259).
> >
> I'm assuming that  "upgrade from 2.2.x to 3.0.2 failed on KVM, if local
> storage is used"  encompasses shared storage as well. We're using GFS.

When you create the storage pool, are you using "sharedmount" or NFS?

> 
> > > -----Original Message-----
> > > From: de Jong, Mark-Jan [mailto:Mark-Jan.deJong@teoco.com]
> > > Sent: Wednesday, June 06, 2012 6:45 PM
> > > To: 'cloudstack-dev@incubator.apache.org'
> > > Subject: RE: 3.0.2 - KVM Agent problem? Please help.
> > >
> > >
> > >
> > > > -----Original Message-----
> > > > From: Edison Su [mailto:Edison.su@citrix.com]
> > > > Sent: Monday, June 04, 2012 5:20 PM
> > > > To: de Jong, Mark-Jan
> > > > Subject: RE: 3.0.2 - KVM Agent problem? Please help.
> > > >
> > > > It's a bug in kvm agent code, I fixed the upgrade issue from
> 2.2.x
> > > > to 3.0.x, but it only worked for shared storage.
> > > > You can fix it by update db with the following sql:
> > > >
> > > > update volumes set path=SUBSTRING_INDEX(`path`, '/', -1);
> > > >
> > > > , then restart mgt server.
> > >
> > > I ran into the same issue and found that, along with the DB update
> > > mentioned above, I had to run the following to fix template VM
> > builds...
> > >
> > > update template_spool_ref set
> > local_path=SUBSTRING_INDEX(`local_path`,
> > > '/', -1);
> > > update template_spool_ref set
> > > install_path=SUBSTRING_INDEX(`install_path`, '/', -1)
> > >
> > >
> > > >
> > > > > -----Original Message-----
> > > > > From: Serg [mailto:kerncore@gmail.com]
> > > > > Sent: Monday, June 04, 2012 1:25 PM
> > > > > To: Edison Su
> > > > > Cc: cloudstack-dev@incubator.apache.org
> > > > > Subject: Re: 3.0.2 - KVM Agent problem? Please help.
> > > > >
> > > > > Hi,
> > > > >
> > > > > Thanks for your response,
> > > > >
> > > > > I turned on debug logging but nothing new has been discovered.
> > > > > after upgrading from 2.2.14 to 3.0.2 all KVM server with Local
> > > > Storage
> > > > > behave identically
> > > > >
> > > > >  - Unable to re-start the instance ( instance can't be started
> > > > anymore
> > > > > )
> > > > >  - Make a snapshot for volume.
> > > > >  - Can't download the volume
> > > > >
> > > > > For now, back to online such instances I'm can by the following
> > > > > way
> > > > > - import volume of stopped instance as template and the start
> new
> > > > > instance from
> > > > >
> > > > > Log :
> > > > > libvir: Storage error : Storage volume not found: no storage
> vol
> > > with
> > > > > matching name
> > > > >
> > > > > [root@ /]# virsh vol-list c6f1f032-5aad-4983-bc83-0ec1a6b5e32c
> |
> > > grep
> > > > > 4341f531-bef6-47b6-ae75-2b8d6de63038
> > > > > 4341f531-bef6-47b6-ae75-2b8d6de63038
> > > > /var/lib/libvirt/images/4341f531-
> > > > > bef6-47b6-ae75-2b8d6de63038
> > > > >
> > > > > [root@ /]# qemu-img check /var/lib/libvirt/images/4341f531-
> bef6-
> > > 47b6-
> > > > > ae75-2b8d6de63038
> > > > > No errors were found on the image.
> > > > >
> > > > > Full log output :
> > > > >
> > > > > Log output : http://pastebin.com/dd0Wz22a
> > > > >
> > > > >
> > > > > --
> > > > > Serg
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Jun 4, 2012, at 8:33 PM, Edison Su wrote:
> > > > >
> > > > > > Please turn log level to debug(/etc/cloud/agent/log4j-
> > cloud.xml)
> > > on
> > > > > your kvm host, then restart cloud-agent, take a look at agent
> log
> > > > again.
> > > > > >
> > > > > >> -----Original Message-----
> > > > > >> From: Serg [mailto:kerncore@gmail.com]
> > > > > >> Sent: Sunday, June 03, 2012 11:48 AM
> > > > > >> To: cloudstack-dev@incubator.apache.org
> > > > > >> Subject: 3.0.2 - KVM Agent problem? Please help.
> > > > > >>
> > > > > >> Hi,
> > > > > >>
> > > > > >>
> > > > > >> How to debug such problem ?
> > > > > >> All volumes exist and instances running, but can't to
> > > > > >>
> > > > > >> CentOS release 6.2 (Final)
> > > > > >> libvirt-0.9.4-23.el6_2.4.x86_64
> > > > > >> libvirt-client-0.9.4-23.el6_2.4.x86_64
> > > > > >>
> > > > > >>
> > > > > >> [root@kvmhost2 snapshot]# tail -10
> > > /var/log/cloud/agent/agent.log
> > > > > >> com.cloud.utils.exception.CloudRuntimeException:
> > > > > >> org.libvirt.LibvirtException: Storage volume not found:
no
> > > storage
> > > > > vol
> > > > > >> with matching name
> > > > > >> '/var/lib/libvirt/images/67033fb9-a432-4162-8b22-
> > > > > >> d8a7b2aeb100'
> > > > > >>  at
> > > > > >>
> > > > >
> > > >
> > >
> >
> com.cloud.agent.storage.LibvirtStorageAdaptor.getVolume(LibvirtStorage
> > > > > A
> > > > > >> daptor.java:81)
> > > > > >> ...cut...
> > > > > >> libvir: Storage error : Storage volume not found: no storage
> > > > > >> vol
> > > > > with
> > > > > >> matching name '/opt/primary/f5b97a37-f8dd-485c-945c-
> > > da31d20b9e23'
> > > > > >> ...cut...
> > > > > >> libvir: Storage error : Storage volume not found: no storage
> > > > > >> vol
> > > > > with
> > > > > >> matching name '/var/lib/libvirt/images/58ba9eed-c864-44f3-
> > a987-
> > > > > >> 9d1eaab182e1'
> > > > > >>
> > > > > >> [root@kvmhost2 snapshot]# tail -10
> > > /var/log/cloud/agent/agent.log
> > > > |
> > > > > awk
> > > > > >> '{print $15}' | xargs file
> > > > > >> /opt/primary/e371ebb7-8d9e-4c64-83e9-effea15a00e7:
> > > Qemu
> > > > > >> Image, Format: Qcow , Version: 2
> > > > > >> /opt/primary/e371ebb7-8d9e-4c64-83e9-effea15a00e7:
> > > Qemu
> > > > > >> Image, Format: Qcow , Version: 2
> > > > > >> /opt/primary/e55358fa-8dc5-4070-b3b6-3f4bc01b17ae:
> > > Qemu
> > > > > >> Image, Format: Qcow , Version: 2
> > > > > >> /opt/primary/e55358fa-8dc5-4070-b3b6-3f4bc01b17ae:
> > > Qemu
> > > > > >> Image, Format: Qcow , Version: 2
> > > > > >> /var/lib/libvirt/images/d49b318c-8bb4-4bb2-a897-1c40ef234d30:
> > > Qemu
> > > > > >> Image, Format: Qcow , Version: 2
> > > > > >> /var/lib/libvirt/images/d49b318c-8bb4-4bb2-a897-1c40ef234d30:
> > > Qemu
> > > > > >> Image, Format: Qcow , Version: 2
> > > > > >> /var/lib/libvirt/images/193c378c-80ba-4b82-affe-45c9da0ee839:
> > > Qemu
> > > > > >> Image, Format: Qcow , Version: 2
> > > > > >> /var/lib/libvirt/images/193c378c-80ba-4b82-affe-45c9da0ee839:
> > > Qemu
> > > > > >> Image, Format: Qcow , Version: 2
> > > > > >> /var/lib/libvirt/images/58ba9eed-c864-44f3-a987-9d1eaab182e1:
> > > Qemu
> > > > > >> Image, Format: Qcow , Version: 2
> > > > > >> /var/lib/libvirt/images/58ba9eed-c864-44f3-a987-9d1eaab182e1:
> > > Qemu
> > > > > >> Image, Format: Qcow , Version: 2
> > > > > >>
> > > > > >> When I'm try to start instance through UI I'm get the
> > following
> > > > > error :
> > > > > >> UI Error pop-up : Resource [Host:34] is unreachable: Host
34:
> > > > > >> Unable
> > > > > to
> > > > > >> start instance due to Unable to get answer that is of class
> > > > > >> com.cloud.agent.api.StartAnswer
> > > > > >>
> > > > > >> Agent LOG :
> > > > > >>
> > > > > >> 2012-06-03 20:47:13,397 WARN
> > > > > >> [resource.computing.LibvirtComputingResource]
> > > > > >> (agentRequest-Handler-
> > > > > >> 2:null) Exception
> > > > > >> com.cloud.utils.exception.CloudRuntimeException:
> > > > > >> org.libvirt.LibvirtException: Storage volume not found:
no
> > > storage
> > > > > vol
> > > > > >> with matching name
> > > > > >> '/var/lib/libvirt/images/19d086dd-8229-48c2-aaa7-
> > > > > >> ea3d218dc5dc'
> > > > > >>  at
> > > > > >>
> > > > >
> > > >
> > >
> >
> com.cloud.agent.storage.LibvirtStorageAdaptor.getVolume(LibvirtStorage
> > > > > A
> > > > > >> daptor.java:81)
> > > > > >>  at
> > > > > >>
> > > > >
> > > >
> > >
> >
> com.cloud.agent.storage.LibvirtStorageAdaptor.getPhysicalDisk(LibvirtS
> > > > > t
> > > > > >> orageAdaptor.java:450)
> > > > > >>  at
> > > > > >>
> > > > >
> > > >
> > >
> >
> com.cloud.agent.storage.LibvirtStoragePool.getPhysicalDisk(LibvirtStor
> > > > > a
> > > > > >> gePool.java:104)
> > > > > >>  at
> > > > > >>
> > > > >
> > > >
> > >
> >
> com.cloud.agent.resource.computing.LibvirtComputingResource.createVbd(
> > > > > L
> > > > > >> ibvirtComputingResource.java:2593)
> > > > > >>  at
> > > > > >>
> > > > >
> > > >
> > >
> >
> com.cloud.agent.resource.computing.LibvirtComputingResource.execute(Li
> > > > > b
> > > > > >> virtComputingResource.java:2510)  at
> > > > > >>
> > > > >
> > > >
> > >
> >
> com.cloud.agent.resource.computing.LibvirtComputingResource.executeReq
> > > > > u
> > > > > >> est(LibvirtComputingResource.java:985)
> > > > > >>  at com.cloud.agent.Agent.processRequest(Agent.java:517)
> > > > > >>  at
> > > > > >>
> > com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:827
> > > > > >> )  at com.cloud.utils.nio.Task.run(Task.java:79)
> > > > > >>  at
> > > > > >>
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.j
> > > > > a
> > > > > >> va:1110)
> > > > > >>  at
> > > > > >>
> > > > >
> > > >
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.
> > > > > j
> > > > > >> ava:603)
> > > > > >>  at java.lang.Thread.run(Thread.java:679)
> > > > > >> 2012-06-03 20:47:13,399{GMT} WARN  [cloud.agent.Agent]
> > > > > (agentRequest-
> > > > > >> Handler-2:) Caught:
> > > > > >> java.lang.NullPointerException
> > > > > >>  at
> > > > > >>
> > > > >
> > > >
> > >
> >
> com.cloud.agent.resource.computing.LibvirtComputingResource.cleanupVMN
> > > > > e
> > > > > >> tworks(LibvirtComputingResource.java:3664)
> > > > > >>  at
> > > > > >>
> > > > >
> > > >
> > >
> >
> com.cloud.agent.resource.computing.LibvirtComputingResource.handleVmSt
> > > > > a
> > > > > >> rtFailure(LibvirtComputingResource.java:2416)
> > > > > >>  at
> > > > > >>
> > > > >
> > > >
> > >
> >
> com.cloud.agent.resource.computing.LibvirtComputingResource.execute(Li
> > > > > b
> > > > > >> virtComputingResource.java:2539)  at
> > > > > >>
> > > > >
> > > >
> > >
> >
> com.cloud.agent.resource.computing.LibvirtComputingResource.executeReq
> > > > > u
> > > > > >> est(LibvirtComputingResource.java:985)
> > > > > >>  at com.cloud.agent.Agent.processRequest(Agent.java:517)
> > > > > >>  at
> > > > > >>
> > com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:827
> > > > > >> )  at com.cloud.utils.nio.Task.run(Task.java:79)
> > > > > >>  at
> > > > > >>
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.j
> > > > > a
> > > > > >> va:1110)
> > > > > >>  at
> > > > > >>
> > > > >
> > > >
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.
> > > > > j
> > > > > >> ava:603)
> > > > > >>  at java.lang.Thread.run(Thread.java:679)
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> --
> > > > > >> Serg
> > > > > >
> > > >
> > >
> > >
> > > PRIVILEGED AND CONFIDENTIAL
> > > PLEASE NOTE: The information contained in this message is
> privileged
> > > and confidential, and is intended only for the use of the
> individual
> > > to whom it is addressed and others who have been specifically
> > > authorized to receive it. If you are not the intended recipient,
> you
> > > are hereby notified that any dissemination, distribution or copying
> > of
> > > this communication is strictly prohibited. If you have received
> this
> > > communication in error, or if any problems occur with transmission,
> > > please contact sender. Thank you.
> > >
> > > Please consider the environment before printing this e-mail.
> 
> 
> PRIVILEGED AND CONFIDENTIAL
> PLEASE NOTE: The information contained in this message is privileged
> and confidential, and is intended only for the use of the individual to
> whom it is addressed and others who have been specifically authorized
> to receive it. If you are not the intended recipient, you are hereby
> notified that any dissemination, distribution or copying of this
> communication is strictly prohibited. If you have received this
> communication in error, or if any problems occur with transmission,
> please contact sender. Thank you.
> 
> Please consider the environment before printing this e-mail.


Mime
View raw message