cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nux! <...@li.nux.ro>
Subject Re: ACS 4.5.1 KVM live migration problem
Date Fri, 15 May 2015 08:30:10 GMT
Andrei,

"trying to fetch storage pool d8d5ec36-3cb0-39af-8fc6-084a4abd5d28 from libvirt", does that
pool actually exist in libvirt?
virsh pool-list --all

What are the libvirtd logs saying?

--
Sent from the Delta quadrant using Borg technology!

Nux!
www.nux.ro

----- Original Message -----
> From: "Andrei Mikhailovsky" <andrei@arhont.com>
> To: users@cloudstack.apache.org
> Sent: Friday, 15 May, 2015 01:04:22
> Subject: Re: ACS 4.5.1 KVM live migration problem

> Lucian,
> 
> I know and I should really have a test env before doing the live update.
> Hopefully will get to it at some point.
> 
> I am running acs 4.5.1 on Ubuntu 14.04 servers with stock qemu and libvirt as
> far as I can tell. The versions are:
> 
> libvirt-bin 1.2.2-0ubuntu13.1.10
> qemu-kvm 2.0.0+dfsg-2ubuntu1.10
> 
> The logs are at the bottom of this email, which i've posted earlier.
> 
> Thanks
> 
> ----- Original Message -----
> 
> From: "Nux!" <nux@li.nux.ro>
> To: users@cloudstack.apache.org
> Sent: Friday, 15 May, 2015 12:10:06 AM
> Subject: Re: ACS 4.5.1 KVM live migration problem
> 
> Andrei,
> 
> It's good to test the RCs, that way anyone can report problems so they don't end
> up in the stable release.
> I am using KVM, alas am using local storage and never test for live migrations
> ...
> 
> Is there any more information you can share? ACS/libvirt/qemu logs etc.
> 
> Lucian
> 
> --
> Sent from the Delta quadrant using Borg technology!
> 
> Nux!
> www.nux.ro
> 
> ----- Original Message -----
>> From: "Andrei Mikhailovsky" <andrei@arhont.com>
>> To: users@cloudstack.apache.org
>> Sent: Thursday, 14 May, 2015 23:47:51
>> Subject: Re: ACS 4.5.1 KVM live migration problem
> 
>> Lucian,
>> 
>> I don't know as i've not tested the 4.5 branch prior to 4.5.1. I know for sure
>> that i've not had any migration issues in 4.4.3, 4.4.2, 4.3.2 and 4.2.1. In
>> fact, I don't recall having any migration issues until i've upgraded to 4.5.1.
>> 
>> Thanks
>> 
>> Andrei
>> 
>> 
>> 
>> ----- Original Message -----
>> 
>> From: "Nux!" <nux@li.nux.ro>
>> To: users@cloudstack.apache.org
>> Sent: Thursday, 14 May, 2015 11:09:51 PM
>> Subject: Re: ACS 4.5.1 KVM live migration problem
>> 
>> Andrei,
>> 
>> Do you know if the problem was also in the RC releases?
>> 
>> Lucian
>> 
>> --
>> Sent from the Delta quadrant using Borg technology!
>> 
>> Nux!
>> www.nux.ro
>> 
>> ----- Original Message -----
>>> From: "Andrei Mikhailovsky" <andrei@arhont.com>
>>> To: users@cloudstack.apache.org
>>> Sent: Thursday, 14 May, 2015 21:37:26
>>> Subject: Re: ACS 4.5.1 KVM live migration problem
>> 
>>> Hi,
>>> 
>>> Does anyone have a clue what is causing the migration issue?
>>> 
>>> Has anyone else using ACS 4.5.1 with KVM and can successfully migrate their
>>> instances?
>>> 
>>> I've not had any migration issues prior to upgrading to version 4.5.1.
>>> 
>>> Thanks
>>> 
>>> ----- Original Message -----
>>> 
>>> From: "Andrei Mikhailovsky" <andrei@arhont.com>
>>> To: users@cloudstack.apache.org
>>> Sent: Wednesday, 13 May, 2015 10:44:29 AM
>>> Subject: Re: ACS 4.5.1 KVM live migration problem
>>> 
>>> Hi Rohit,
>>> 
>>> forgot to answer you on the cloud.vlan table.
>>> 
>>> That particular vm has a network with vlan id 1151 as shown when i look at the
>>> network details in the acs gui. However, this vlan is not shown in the
>>> cloud.vlan table. From what I can see the cloud.vlan table shows only the
>>> public and management network vlan interfaces and does not show the guest
>>> network vlans.
>>> 
>>> In terms of the public network vlan which is used for routing traffic to the
>>> internet from this particular vm, it is:
>>> 
>>> 
>>> mysql> select * from vlan where id=12;
>>> +----+--------------------------------------+-------------+---------------+-----------------+-------------------------------+----------------+----------------+------------+---------------------+-------------+----------+-----------+---------+---------+
>>>| id | uuid | vlan_id | vlan_gateway | vlan_netmask | description | vlan_type
|
>>>| data_center_id | network_id | physical_network_id | ip6_gateway | ip6_cidr |
>>>| ip6_range | removed | created |
>>> +----+--------------------------------------+-------------+---------------+-----------------+-------------------------------+----------------+----------------+------------+---------------------+-------------+----------+-----------+---------+---------+
>>>| 12 | d13ea4b3-2087-4376-9d0a-f54efe2a55af | vlan://2030 | 178.XXX.XXX.1 |
>>>| 255.255.255.128 | 178.XXX.XXX.2-178.XXX.XXX.119 | VirtualNetwork | 1 | 200 |
>>>| 200 | NULL | NULL | NULL | NULL | NULL |
>>> +----+--------------------------------------+-------------+---------------+-----------------+-------------------------------+----------------+----------------+------------+---------------------+-------------+----------+-----------+---------+---------+
>>> 1 row in set (0.00 sec)
>>> 
>>> 
>>> Hope that helps
>>> 
>>> Andrei
>>> ----- Original Message -----
>>> 
>>> From: "Rohit Yadav" <rohit.yadav@shapeblue.com>
>>> To: users@cloudstack.apache.org
>>> Sent: Wednesday, 13 May, 2015 8:55:55 AM
>>> Subject: Re: ACS 4.5.1 KVM live migration problem
>>> 
>>> Hi Andrei,
>>> 
>>> This looks like an issue similar to
>>> https://issues.apache.org/jira/browse/CLOUDSTACK-6893
>>> Can share the row from your cloud.vlan table and value of “select cache_mode
>>> from volume_view where vm_id=<put the vm id here>\G;" for the VM causing
the
>>> NPE?
>>> 
>>>> On 12-May-2015, at 10:51 pm, Andrei Mikhailovsky <andrei@arhont.com>
wrote:
>>>> 
>>>> 
>>>> 
>>>> It seems that the problem is worse than i've initially thought. In fact,
I can't
>>>> migrate most of my vms apart from a handful and I can't determine a correlation
>>>> between the migrateable vms and once that produce exception.
>>>> 
>>>> Thanks for any help.
>>>> 
>>>> Andrei
>>>> 
>>>> ----- Original Message -----
>>>> 
>>>> From: "Andrei Mikhailovsky" <andrei@arhont.com>
>>>> To: users@cloudstack.apache.org
>>>> Sent: Tuesday, 12 May, 2015 8:53:16 PM
>>>> Subject: ACS 4.5.1 KVM live migration problem
>>>> 
>>>> Hi,
>>>> 
>>>> I am having an issue migrating some of vms after recently upgrading to ACS
>>>> 4.5.1. I am running Ubuntu 14.04 on both host and management servers. Here
is
>>>> the output from the log file on a client agent :
>>>> 
>>>> 
>>>> 2015-05-12 20:42:34,154 DEBUG [kvm.resource.LibvirtComputingResource]
>>>> (agentRequest-Handler-1:null) Preparing host for migrating
>>>> com.cloud.agent.api.to.VirtualMachineTO@21a038ac
>>>> 2015-05-12 20:42:34,157 DEBUG [kvm.resource.LibvirtConnection]
>>>> (agentRequest-Handler-1:null) can't find connection: KVM, for vm: i-9-1162-VM,
>>>> continue
>>>> 2015-05-12 20:42:34,159 DEBUG [kvm.resource.LibvirtConnection]
>>>> (agentRequest-Handler-1:null) can't find connection: LXC, for vm: i-9-1162-VM,
>>>> continue
>>>> 2015-05-12 20:42:34,159 DEBUG [kvm.resource.LibvirtConnection]
>>>> (agentRequest-Handler-1:null) can't find which hypervisor the vm used , then
>>>> use the default hypervisor
>>>> 2015-05-12 20:42:34,160 DEBUG [kvm.resource.BridgeVifDriver]
>>>> (agentRequest-Handler-1:null) nic=[Nic:Guest-178.248.108.205-vlan://2014]
>>>> 2015-05-12 20:42:34,160 DEBUG [kvm.resource.BridgeVifDriver]
>>>> (agentRequest-Handler-1:null) creating a vNet dev and bridge for guest traffic
>>>> per traffic label cloudstackbr0
>>>> 2015-05-12 20:42:34,160 DEBUG [kvm.resource.BridgeVifDriver]
>>>> (agentRequest-Handler-1:null) Executing:
>>>> /usr/share/cloudstack-common/scripts/vm/network/vnet/modifyvlan.sh -v 2014
-p
>>>> bond0 -b brbond0-2014 -o add
>>>> 2015-05-12 20:42:34,211 DEBUG [kvm.resource.BridgeVifDriver]
>>>> (agentRequest-Handler-1:null) Execution is successful.
>>>> 2015-05-12 20:42:34,211 DEBUG [kvm.resource.BridgeVifDriver]
>>>> (agentRequest-Handler-1:null) nic=[Nic:Guest-10.1.1.66-null]
>>>> 2015-05-12 20:42:34,212 DEBUG [kvm.storage.KVMStoragePoolManager]
>>>> (agentRequest-Handler-1:null) Disconnecting disk
>>>> 23add201-e4ee-447b-a448-ecd152aea4ad
>>>> 2015-05-12 20:42:34,212 DEBUG [kvm.storage.LibvirtStorageAdaptor]
>>>> (agentRequest-Handler-1:null) Trying to fetch storage pool
>>>> cf771bc7-8998-354d-8e10-5564585a3c20 from libvirt
>>>> 2015-05-12 20:42:34,223 DEBUG [kvm.storage.KVMStoragePoolManager]
>>>> (agentRequest-Handler-1:null) Disconnecting disk
>>>> 55100d25-410e-4fa3-a38b-7717f74d2afe
>>>> 2015-05-12 20:42:34,223 DEBUG [kvm.storage.LibvirtStorageAdaptor]
>>>> (agentRequest-Handler-1:null) Trying to fetch storage pool
>>>> cf771bc7-8998-354d-8e10-5564585a3c20 from libvirt
>>>> 2015-05-12 20:42:34,232 DEBUG [kvm.storage.KVMStoragePoolManager]
>>>> (agentRequest-Handler-1:null) Disconnecting disk
>>>> 2db59d16-d17f-49a1-b913-7fbe4025a549
>>>> 2015-05-12 20:42:34,233 DEBUG [kvm.storage.LibvirtStorageAdaptor]
>>>> (agentRequest-Handler-1:null) Trying to fetch storage pool
>>>> cf771bc7-8998-354d-8e10-5564585a3c20 from libvirt
>>>> 2015-05-12 20:42:34,243 DEBUG [kvm.storage.KVMStoragePoolManager]
>>>> (agentRequest-Handler-1:null) Disconnecting disk
>>>> 17afbf31-ac89-46f7-a2c8-f8aed796e4c6
>>>> 2015-05-12 20:42:34,243 DEBUG [kvm.storage.LibvirtStorageAdaptor]
>>>> (agentRequest-Handler-1:null) Trying to fetch storage pool
>>>> d8d5ec36-3cb0-39af-8fc6-084a4abd5d28 from libvirt
>>>> 2015-05-12 20:42:34,254 WARN [cloud.agent.Agent] (agentRequest-Handler-1:null)
>>>> Caught:
>>>> java.lang.NullPointerException
>>>> at
>>>> com.cloud.network.Networks$BroadcastDomainType.getSchemeValue(Networks.java:172)
>>>> at com.cloud.network.Networks$BroadcastDomainType.getValue(Networks.java:226)
>>>> at
>>>> com.cloud.hypervisor.kvm.resource.BridgeVifDriver.plug(BridgeVifDriver.java:105)
>>>> at
>>>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.execute(LibvirtComputingResource.java:3230)
>>>> at
>>>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1307)
>>>> at com.cloud.agent.Agent.processRequest(Agent.java:503)
>>>> at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:808)
>>>> at com.cloud.utils.nio.Task.run(Task.java:84)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 2015-05-12 20:42:34,256 DEBUG [cloud.agent.Agent] (agentRequest-Handler-1:null)
>>>> Seq 7-7525233502359390941: { Ans: , MgmtId: 115129173025118, via: 7, Ver:
v1,
>>>> Flags: 110,
>>>> [{"com.cloud.agent.api.Answer":{"result":false,"details":"java.lang.NullPointerException\n\tat
>>>> com.cloud.network.Networks$BroadcastDomainType.getSchemeValue(Networks.java:172)\n\tat
>>>> com.cloud.network.Networks$BroadcastDomainType.getValue(Networks.java:226)\n\tat
>>>> com.cloud.hypervisor.kvm.resource.BridgeVifDriver.plug(BridgeVifDriver.java:105)\n\tat
>>>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.execute(LibvirtComputingResource.java:3230)\n\tat
>>>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1307)\n\tat
>>>> com.cloud.agent.Agent.processRequest(Agent.java:503)\n\tat
>>>> com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:808)\n\tat
>>>> com.cloud.utils.nio.Task.run(Task.java:84)\n\tat
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\n\tat
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\n\tat
>>>> java.lang.Thread.run(Thread.java:745)\n","wait":0}}] }
>>>> 
>>>> 
>>>> 
>>>> Any idea how to get this fixed? Not sure why all of a sudden the migration
>>>> stopped working for a handful of vms. I can successfully migrate some vms,
but
>>>> not others.
>>>> 
>>>> Thanks
>>>> 
>>>> Andrei
>>>> 
>>>> 
>>> 
>>> Regards,
>>> Rohit Yadav
>>> Software Architect, ShapeBlue
>>> M. +91 88 262 30892 | rohit.yadav@shapeblue.com
>>> Blog: bhaisaab.org | Twitter: @_bhaisaab
>>> 
>>> 
>>> 
>>> Find out more about ShapeBlue and our range of CloudStack related services
>>> 
>>> IaaS Cloud Design & Build<http://shapeblue.com/iaas-cloud-design-and-build//>
>>> CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/>
>>> CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/>
>>> CloudStack Software
>>> Engineering<http://shapeblue.com/cloudstack-software-engineering/>
>>> CloudStack Infrastructure
>>> Support<http://shapeblue.com/cloudstack-infrastructure-support/>
>>> CloudStack Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>
>>> 
>>> This email and any attachments to it may be confidential and are intended solely
>>> for the use of the individual to whom it is addressed. Any views or opinions
>>> expressed are solely those of the author and do not necessarily represent those
>>> of Shape Blue Ltd or related companies. If you are not the intended recipient
>>> of this email, you must neither take any action based upon its contents, nor
>>> copy or show it to anyone. Please contact the sender if you believe you have
>>> received this email in error. Shape Blue Ltd is a company incorporated in
>>> England & Wales. ShapeBlue Services India LLP is a company incorporated in
>>> India and is operated under license from Shape Blue Ltd. Shape Blue Brasil
>>> Consultoria Ltda is a company incorporated in Brasil and is operated under
>>> license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by
>>> The Republic of South Africa and is traded under license from Shape Blue Ltd.
> > > ShapeBlue is a registered trademark.

Mime
View raw message