cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Mabry <dma...@ena.com.INVALID>
Subject Re: CS 4.8 KVM VMs will not live migrate
Date Mon, 29 Jan 2018 23:41:59 GMT
Mike,

Thanks for the reply.  As requested:

Will not Migrate
============================
<domain type='kvm' id='63'>
  <name>i-532-1392-NSVLTN</name>
  <uuid>f7dbf00b-2e15-4991-a407-cf27a3d65d1e</uuid>
  <description>Other PV Virtio-SCSI (64-bit)</description>
  <memory unit='KiB'>4194304</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <vcpu placement='static'>2</vcpu>
  <cputune>
    <shares>2000</shares>
  </cputune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <sysinfo type='smbios'>
    <system>
      <entry name='manufacturer'>Apache Software Foundation</entry>
      <entry name='product'>CloudStack KVM Hypervisor</entry>
      <entry name='uuid'>f7dbf00b-2e15-4991-a407-cf27a3d65d1e</entry>
    </system>
  </sysinfo>
  <os>
    <type arch='x86_64' machine='pc-i440fx-rhel7.0.0'>hvm</type>
    <boot dev='cdrom'/>
    <boot dev='hd'/>
    <smbios mode='sysinfo'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <cpu mode='custom' match='exact' check='partial'>
    <model fallback='allow'>Haswell-noTSX</model>
    <feature policy='require' name='erms'/>
  </cpu>
  <clock offset='utc'>
    <timer name='kvmclock'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='network' device='disk'>
      <driver name='qemu' type='raw' cache='none' discard='unmap'/>
      <auth username='c01z01'>
        <secret type='ceph' uuid='10a32867-3386-369e-a391-c8e05e0fa8ed'/>
      </auth>
      <source protocol='rbd' name='c01z01/223e08b0-929c-4c47-833d-1f1de48610f3'>
        <host name='cephmonc01.nsvltn.ena.net' port='6789'/>
      </source>
      <backingStore/>
      <target dev='sda' bus='scsi'/>
      <iotune>
        <read_bytes_sec>524288000</read_bytes_sec>
        <write_bytes_sec>524288000</write_bytes_sec>
        <read_iops_sec>500</read_iops_sec>
        <write_iops_sec>500</write_iops_sec>
      </iotune>
      <serial>223e08b0929c4c47833d</serial>
      <alias name='scsi0-0-0-0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='network' device='disk'>
      <driver name='qemu' type='raw' cache='none' discard='unmap'/>
      <auth username='c01z01'>
        <secret type='ceph' uuid='10a32867-3386-369e-a391-c8e05e0fa8ed'/>
      </auth>
      <source protocol='rbd' name='c01z01/97e5a299-1efd-40ed-85f4-a5c365a56414'>
        <host name='cephmonc01.nsvltn.ena.net' port='6789'/>
      </source>
      <backingStore/>
      <target dev='sdb' bus='scsi'/>
      <iotune>
        <read_bytes_sec>524288000</read_bytes_sec>
        <write_bytes_sec>524288000</write_bytes_sec>
        <read_iops_sec>500</read_iops_sec>
        <write_iops_sec>500</write_iops_sec>
      </iotune>
      <serial>97e5a2991efd40ed85f4</serial>
      <alias name='scsi0-0-0-1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw' cache='none'/>
      <backingStore/>
      <target dev='hdc' bus='ide'/>
      <readonly/>
      <alias name='ide0-1-0'/>
      <address type='drive' controller='0' bus='1' target='0' unit='0'/>
    </disk>
    <controller type='scsi' index='0'>
      <alias name='scsi0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </controller>
    <controller type='usb' index='0' model='piix3-uhci'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'>
      <alias name='pci.0'/>
    </controller>
    <controller type='ide' index='0'>
      <alias name='ide'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <interface type='bridge'>
      <mac address='02:00:08:da:00:1d'/>
      <source bridge='brvx-5312'/>
      <bandwidth>
        <inbound average='128000' peak='128000'/>
        <outbound average='128000' peak='128000'/>
      </bandwidth>
      <target dev='vnet32'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/27'/>
      <target port='0'/>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/27'>
      <source path='/dev/pts/27'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <input type='tablet' bus='usb'>
      <alias name='input0'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='5927' autoport='yes' listen='10.10.10.10'>
      <listen type='address' address='10.10.10.10'/>
    </graphics>
    <video>
      <model type='cirrus' vram='16384' heads='1' primary='yes'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <memballoon model='none'/>
  </devices>
</domain>

Will migrate
============================
<domain type='kvm' id='24'>
  <name>i-532-1298-NSVLTN</name>
  <uuid>d6ec74b8-4f6a-405c-834e-ece42151b802</uuid>
  <description>Windows PV</description>
  <memory unit='KiB'>4194304</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <vcpu placement='static'>1</vcpu>
  <cputune>
    <shares>1000</shares>
  </cputune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <sysinfo type='smbios'>
    <system>
      <entry name='manufacturer'>Apache Software Foundation</entry>
      <entry name='product'>CloudStack KVM Hypervisor</entry>
      <entry name='uuid'>d6ec74b8-4f6a-405c-834e-ece42151b802</entry>
    </system>
  </sysinfo>
  <os>
    <type arch='x86_64' machine='pc-i440fx-rhel7.0.0'>hvm</type>
    <boot dev='cdrom'/>
    <boot dev='hd'/>
    <smbios mode='sysinfo'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>Haswell</model>
    <feature policy='disable' name='hle'/>
    <feature policy='disable' name='rtm'/>
    <feature policy='require' name='erms'/>
    <feature policy='require' name='hypervisor'/>
    <feature policy='require' name='xsaveopt'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='network' device='disk'>
      <driver name='qemu' type='raw' cache='none'/>
      <auth username='c01z01'>
        <secret type='ceph' uuid='10a32867-3386-369e-a391-c8e05e0fa8ed'/>
      </auth>
      <source protocol='rbd' name='c01z01/f0b58e22-d05a-4825-8a4a-df0003a261b0'>
        <host name='cephmonc01.nsvltn.ena.net' port='6789'/>
      </source>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <iotune>
        <read_bytes_sec>524288000</read_bytes_sec>
        <write_bytes_sec>524288000</write_bytes_sec>
        <read_iops_sec>500</read_iops_sec>
        <write_iops_sec>500</write_iops_sec>
      </iotune>
      <serial>f0b58e22d05a48258a4a</serial>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>
    <disk type='network' device='disk'>
      <driver name='qemu' type='raw' cache='none'/>
      <auth username='c01z01'>
        <secret type='ceph' uuid='10a32867-3386-369e-a391-c8e05e0fa8ed'/>
      </auth>
      <source protocol='rbd' name='c01z01/cd0c2822-3912-4730-ac55-2c1b4d99ab1d'>
        <host name='cephmonc01.nsvltn.ena.net' port='6789'/>
      </source>
      <backingStore/>
      <target dev='vdb' bus='virtio'/>
      <iotune>
        <read_bytes_sec>524288000</read_bytes_sec>
        <write_bytes_sec>524288000</write_bytes_sec>
        <read_iops_sec>500</read_iops_sec>
        <write_iops_sec>500</write_iops_sec>
      </iotune>
      <serial>cd0c282239124730ac55</serial>
      <alias name='virtio-disk1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw' cache='none'/>
      <backingStore/>
      <target dev='hdc' bus='ide'/>
      <readonly/>
      <alias name='ide0-1-0'/>
      <address type='drive' controller='0' bus='1' target='0' unit='0'/>
    </disk>
    <controller type='ide' index='0'>
      <alias name='ide'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='piix3-uhci'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'>
      <alias name='pci.0'/>
    </controller>
    <interface type='bridge'>
      <mac address='02:00:68:b4:00:0d'/>
      <source bridge='brvx-5312'/>
      <bandwidth>
        <inbound average='128000' peak='128000'/>
        <outbound average='128000' peak='128000'/>
      </bandwidth>
      <target dev='vnet19'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/21'/>
      <target port='0'/>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/21'>
      <source path='/dev/pts/21'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <input type='tablet' bus='usb'>
      <alias name='input0'/>
    </input>
    <input type='mouse' bus='ps2'>
      <alias name='input1'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input2'/>
    </input>
    <graphics type='vnc' port='5919' autoport='yes' listen='10.10.10.10'>
      <listen type='address' address='10.10.10.10'/>
    </graphics>
    <video>
      <model type='cirrus' vram='16384' heads='1' primary='yes'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <memballoon model='none'/>
  </devices>
  <seclabel type='none' model='none'/>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+107:+107</label>
    <imagelabel>+107:+107</imagelabel>
  </seclabel>
</domain>


David Mabry
Manager of Systems Engineering
On 1/29/18, 5:30 PM, "Tutkowski, Mike" <Mike.Tutkowski@netapp.com> wrote:

    Hi David,
    
    So, I don’t know if what I am going to say here will at all be of use to you, but maybe.
:)
    
    I had a customer one time mention to me that he had trouble with live VM migration on
KVM with a VM that was created on an older version of CloudStack. Live VM migration worked
fine for these VMs on the older version of CloudStack (I think it was version 4.5) and stopped
working when he upgraded to 4.8. New VMs (VMs created on the newer version of CloudStack)
worked fine for this feature on 4.8, but old VMs had to be stopped and re-started for live
VM migration to work. I believe the older version of CloudStack was not placing the serial
number of the VM in the VM’s XML descriptor file, but newer versions of CloudStack were
expecting this field.
    
    Can you dump the XML of one or both of your VMs that don’t live migrate and see if they
have the serial number field in their XML? Then, I’d recommend dumping the XML of the VM
that works and seeing if it does, in fact, have the serial number field in its XML.
    
    I hope this is of some help.
    
    Talk to you later,
    Mike
    
    On 1/29/18, 11:48 AM, "David Mabry" <dmabry@ena.com.INVALID> wrote:
    
        Good day Cloudstack Devs,
        
        I've run across a real head scratcher.  I have two VMs, (initially 3 VMs, but more
on that later) on a single host, that I cannot live migrate to any other host in the same
cluster.  We discovered this after attempting to roll out patches going from CentOS 7.2 to
CentOS 7.4.  Initially, we thought it had something to do with the new version of libvirtd
or qemu-kvm on the other hosts in the cluster preventing these VMs from migrating, but we
are able to live migrate other VMs to and from this host without issue.  We can even create
new VMs on this specific host and live migrate them after creation with no issue.  We've put
the migration source agent, migration destination agent and the management server in debug
and don't seem to get anything useful other than "Unsupported command".  Luckily, we did have
one VM that was shutdown and restarted, this is the 3rd VM mentioned above.  Since that VM
has been restarted, it has no issues live migrating to any other host in the cluster.
        
        I'm at a loss as to what to try next and I'm hoping that someone out there might have
had a similar issue and could shed some light on what to do.  Obviously, I can contact the
customer and have them shutdown their VMs, but that will potentially just delay this problem
to be solved another day.  Even if shutting down the VMs is ultimately the solution, I'd still
like to understand what happened to cause this issue in the first place with the hopes of
preventing it in the future.
        
        Here's some information about my setup:
        Cloudstack 4.8 Advanced Networking
        CentOS 7.2 and 7.4 Hosts
        Ceph RBD Primary Storage
        NFS Secondary Storage
        Instance in Question for Debug: i-532-1392-NSVLTN
        
        I have attached relevant debug logs to this email if anyone wishes to take a look.
 I think the most interesting error message that I have received is the following:
        
        468390:2018-01-27 08:59:35,172 DEBUG [c.c.a.t.Request] (Work-Job-Executor-6:ctx-188ea30f
job-181792/job-181802 ctx-8e7f45ad) (logid:f0888362) Seq 22-942378222027276319: Received:
 { Ans: , MgmtId: 14038012703634, via: 22(csh02c01z01.nsvltn.ena.net), Ver: v1, Flags: 110,
{ UnsupportedAnswer } }
        468391:2018-01-27 08:59:35,172 WARN  [c.c.a.m.AgentManagerImpl] (Work-Job-Executor-6:ctx-188ea30f
job-181792/job-181802 ctx-8e7f45ad) (logid:f0888362) Unsupported Command: Unsupported command
issued: com.cloud.agent.api.PrepareForMigrationCommand.  Are you sure you got the right type
of server?
        468392:2018-01-27 08:59:35,179 ERROR [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-6:ctx-188ea30f
job-181792/job-181802 ctx-8e7f45ad) (logid:f0888362) Invocation exception, caused by: com.cloud.exception.AgentUnavailableException:
Resource [Host:22] is unreachable: Host 22: Unable to prepare for migration due to Unsupported
command issued: com.cloud.agent.api.PrepareForMigrationCommand.  Are you sure you got the
right type of server?
        468393:2018-01-27 08:59:35,179 INFO  [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-6:ctx-188ea30f
job-181792/job-181802 ctx-8e7f45ad) (logid:f0888362) Rethrow exception com.cloud.exception.AgentUnavailableException:
Resource [Host:22] is unreachable: Host 22: Unable to prepare for migration due to Unsupported
command issued: com.cloud.agent.api.PrepareForMigrationCommand.  Are you sure you got the
right type of server?
        
        I've tracked this "Unsupported command" down in the CS 4.8 code to cloudstack/api/src/com/cloud/agent/api/Answer.java
which is the generic answer class.  I believe where the error is really being spawned from
is cloudstack/engine/orchestration/src/com/cloud/vm/VirtualMachineManagerImpl.java.  Specifically:
                Answer pfma = null;
                try {
                    pfma = _agentMgr.send(dstHostId, pfmc);
                    if (pfma == null || !pfma.getResult()) {
                        final String details = pfma != null ? pfma.getDetails() : "null answer
returned";
                        final String msg = "Unable to prepare for migration due to " + details;
                        pfma = null;
                        throw new AgentUnavailableException(msg, dstHostId);
                    }
        
        The pfma returned must be in error or is never returned and therefore still null.
 That answer appears that it should be coming from the destination agent, but for the life
of me I can't figure out what the root cause of this error is beyond, "Unsupported command
issued".  What command is unsupported?  My guess is that it could be something wrong with
the dxml that is generated and passed to the destination host, but I have as yet been unable
to catch that dxml in debug.
        
        Any help or guidance is greatly appreciated.
        
        Thanks,
        David Mabry
        
        
    
    

Mime
View raw message