cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tutkowski, Mike" <Mike.Tutkow...@netapp.com>
Subject Re: CS 4.8 KVM VMs will not live migrate
Date Mon, 29 Jan 2018 20:26:55 GMT
Hi David,

So, I don’t know if what I am going to say here will at all be of use to you, but maybe.
:)

I had a customer one time mention to me that he had trouble with live VM migration on KVM
with a VM that was created on an older version of CloudStack. Live VM migration worked fine
for these VMs on the older version of CloudStack (I think it was version 4.5) and stopped
working when he upgraded to 4.8. New VMs (VMs created on the newer version of CloudStack)
worked fine for this feature on 4.8, but old VMs had to be stopped and re-started for live
VM migration to work. I believe the older version of CloudStack was not placing the serial
number of the VM in the VM’s XML descriptor file, but newer versions of CloudStack were
expecting this field.

Can you dump the XML of one or both of your VMs that don’t live migrate and see if they
have the serial number field in their XML? Then, I’d recommend dumping the XML of the VM
that works and seeing if it does, in fact, have the serial number field in its XML.

I hope this is of some help.

Talk to you later,
Mike

On 1/29/18, 11:48 AM, "David Mabry" <dmabry@ena.com.INVALID> wrote:

    Good day Cloudstack Devs,
    
    I've run across a real head scratcher.  I have two VMs, (initially 3 VMs, but more on
that later) on a single host, that I cannot live migrate to any other host in the same cluster.
 We discovered this after attempting to roll out patches going from CentOS 7.2 to CentOS 7.4.
 Initially, we thought it had something to do with the new version of libvirtd or qemu-kvm
on the other hosts in the cluster preventing these VMs from migrating, but we are able to
live migrate other VMs to and from this host without issue.  We can even create new VMs on
this specific host and live migrate them after creation with no issue.  We've put the migration
source agent, migration destination agent and the management server in debug and don't seem
to get anything useful other than "Unsupported command".  Luckily, we did have one VM that
was shutdown and restarted, this is the 3rd VM mentioned above.  Since that VM has been restarted,
it has no issues live migrating to any other host in the cluster.
    
    I'm at a loss as to what to try next and I'm hoping that someone out there might have
had a similar issue and could shed some light on what to do.  Obviously, I can contact the
customer and have them shutdown their VMs, but that will potentially just delay this problem
to be solved another day.  Even if shutting down the VMs is ultimately the solution, I'd still
like to understand what happened to cause this issue in the first place with the hopes of
preventing it in the future.
    
    Here's some information about my setup:
    Cloudstack 4.8 Advanced Networking
    CentOS 7.2 and 7.4 Hosts
    Ceph RBD Primary Storage
    NFS Secondary Storage
    Instance in Question for Debug: i-532-1392-NSVLTN
    
    I have attached relevant debug logs to this email if anyone wishes to take a look.  I
think the most interesting error message that I have received is the following:
    
    468390:2018-01-27 08:59:35,172 DEBUG [c.c.a.t.Request] (Work-Job-Executor-6:ctx-188ea30f
job-181792/job-181802 ctx-8e7f45ad) (logid:f0888362) Seq 22-942378222027276319: Received:
 { Ans: , MgmtId: 14038012703634, via: 22(csh02c01z01.nsvltn.ena.net), Ver: v1, Flags: 110,
{ UnsupportedAnswer } }
    468391:2018-01-27 08:59:35,172 WARN  [c.c.a.m.AgentManagerImpl] (Work-Job-Executor-6:ctx-188ea30f
job-181792/job-181802 ctx-8e7f45ad) (logid:f0888362) Unsupported Command: Unsupported command
issued: com.cloud.agent.api.PrepareForMigrationCommand.  Are you sure you got the right type
of server?
    468392:2018-01-27 08:59:35,179 ERROR [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-6:ctx-188ea30f
job-181792/job-181802 ctx-8e7f45ad) (logid:f0888362) Invocation exception, caused by: com.cloud.exception.AgentUnavailableException:
Resource [Host:22] is unreachable: Host 22: Unable to prepare for migration due to Unsupported
command issued: com.cloud.agent.api.PrepareForMigrationCommand.  Are you sure you got the
right type of server?
    468393:2018-01-27 08:59:35,179 INFO  [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-6:ctx-188ea30f
job-181792/job-181802 ctx-8e7f45ad) (logid:f0888362) Rethrow exception com.cloud.exception.AgentUnavailableException:
Resource [Host:22] is unreachable: Host 22: Unable to prepare for migration due to Unsupported
command issued: com.cloud.agent.api.PrepareForMigrationCommand.  Are you sure you got the
right type of server?
    
    I've tracked this "Unsupported command" down in the CS 4.8 code to cloudstack/api/src/com/cloud/agent/api/Answer.java
which is the generic answer class.  I believe where the error is really being spawned from
is cloudstack/engine/orchestration/src/com/cloud/vm/VirtualMachineManagerImpl.java.  Specifically:
            Answer pfma = null;
            try {
                pfma = _agentMgr.send(dstHostId, pfmc);
                if (pfma == null || !pfma.getResult()) {
                    final String details = pfma != null ? pfma.getDetails() : "null answer
returned";
                    final String msg = "Unable to prepare for migration due to " + details;
                    pfma = null;
                    throw new AgentUnavailableException(msg, dstHostId);
                }
    
    The pfma returned must be in error or is never returned and therefore still null.  That
answer appears that it should be coming from the destination agent, but for the life of me
I can't figure out what the root cause of this error is beyond, "Unsupported command issued".
 What command is unsupported?  My guess is that it could be something wrong with the dxml
that is generated and passed to the destination host, but I have as yet been unable to catch
that dxml in debug.
    
    Any help or guidance is greatly appreciated.
    
    Thanks,
    David Mabry
    
    

Mime
View raw message