cloudstack-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sanjeev N (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (CLOUDSTACK-5745) [Hyper-v] System vms live migration does not work
Date Fri, 03 Jan 2014 06:16:53 GMT

     [ https://issues.apache.org/jira/browse/CLOUDSTACK-5745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Sanjeev N updated CLOUDSTACK-5745:
----------------------------------

    Attachment: management-server.rar

Attached management server log file.

> [Hyper-v] System vms live migration does not work
> -------------------------------------------------
>
>                 Key: CLOUDSTACK-5745
>                 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-5745
>             Project: CloudStack
>          Issue Type: Bug
>      Security Level: Public(Anyone can view this level - this is the default.) 
>          Components: Hypervisor Controller
>    Affects Versions: 4.3.0
>         Environment: Latest build from 4.3 branch with commit:
> 95b6a7b96dab1c77e25987d6fe6003b4447281f1
>            Reporter: Sanjeev N
>            Priority: Critical
>              Labels: hyper-V,, hyper-v, hyperv
>             Fix For: 4.3.0
>
>         Attachments: management-server.rar
>
>
> [Hyper-v] System vms live migration does not work
> Steps to Reproduce:
> ================
> 1.Bring up CS in advanced zone with at-least two Hyperv hosts in a cluster
> 2.Deploy couple of guest vms.
> 3.Try live migrating any of the system vms (SSVM,CPVM,VR)
> Result:
> ======
> Live migration of system vms failes with NPE. Guest vms live migration works fine.
> Following are the sequence of events during live migrations:
> 2014-01-03 11:31:05,427 DEBUG [c.c.a.ApiServlet] (catalina-exec-22:ctx-71773ac9) ===START===
 10.146.0.133 -- GET  command=migrateSystemVm&hostid=a138433d-0aa6-4711-9559-9738ea7e03fc&virtualmachineid=df65d272-b64f-4b5e-9def-e0b34aa481e6&response=json&sessionkey=48pxYt3vmMxm4yn7pwbEszyG7V4%3D&_=1388728864662
> 2014-01-03 11:31:05,570 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (catalina-exec-22:ctx-71773ac9
ctx-0b17c466) submit async job-47, details: AsyncJobVO {id:47, userId: 2, accountId: 2, instanceType:
None, instanceId: null, cmd: org.apache.cloudstack.api.command.admin.systemvm.MigrateSystemVMCmd,
cmdInfo: {"response":"json","sessionkey":"48pxYt3vmMxm4yn7pwbEszyG7V4\u003d","virtualmachineid":"df65d272-b64f-4b5e-9def-e0b34aa481e6","cmdEventType":"VM.MIGRATE","hostid":"a138433d-0aa6-4711-9559-9738ea7e03fc","ctxUserId":"2","httpmethod":"GET","_":"1388728864662","ctxAccountId":"2","ctxStartEventId":"109"},
cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid:
132129494109518, completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
> 2014-01-03 11:31:05,572 INFO  [o.a.c.f.j.i.AsyncJobMonitor] (Job-Executor-57:ctx-f75b2d46)
Add job-47 into job monitoring
> 2014-01-03 11:31:05,572 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Job-Executor-57:ctx-f75b2d46)
Executing AsyncJobVO {id:47, userId: 2, accountId: 2, instanceType: None, instanceId: null,
cmd: org.apache.cloudstack.api.command.admin.systemvm.MigrateSystemVMCmd, cmdInfo: {"response":"json","sessionkey":"48pxYt3vmMxm4yn7pwbEszyG7V4\u003d","virtualmachineid":"df65d272-b64f-4b5e-9def-e0b34aa481e6","cmdEventType":"VM.MIGRATE","hostid":"a138433d-0aa6-4711-9559-9738ea7e03fc","ctxUserId":"2","httpmethod":"GET","_":"1388728864662","ctxAccountId":"2","ctxStartEventId":"109"},
cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid:
132129494109518, completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
> 2014-01-03 11:31:05,572 DEBUG [c.c.a.ApiServlet] (catalina-exec-22:ctx-71773ac9 ctx-0b17c466)
===END===  10.146.0.133 -- GET  command=migrateSystemVm&hostid=a138433d-0aa6-4711-9559-9738ea7e03fc&virtualmachineid=df65d272-b64f-4b5e-9def-e0b34aa481e6&response=json&sessionkey=48pxYt3vmMxm4yn7pwbEszyG7V4%3D&_=1388728864662
> 2014-01-03 11:31:05,682 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Job-Executor-57:ctx-f75b2d46
ctx-0b17c466) Sync job-48 execution on object VmWorkJobQueue.4
> 2014-01-03 11:31:05,778 DEBUG [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-3:null)
SeqA 4-10358: Processing Seq 4-10358:  { Cmd , MgmtId: -1, via: 4, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":2,"_loadInfo":"{\n
 \"connections\": [\n    {\n      \"id\": 19,\n      \"clientInfo\": \"\",\n      \"host\":
\"C9FB9529-1A5D-494A-822B-068D485F888B\",\n      \"port\": 2179,\n      \"tag\": \"df65d272-b64f-4b5e-9def-e0b34aa481e6\",\n
     \"createTime\": 1388728599282,\n      \"lastUsedTime\": 1388728719852\n    },\n    {\n
     \"id\": 18,\n      \"clientInfo\": \"\",\n      \"host\": \"B5FCE673-EA38-43C0-A130-F7FB5E40FF49\",\n
     \"port\": 2179,\n      \"tag\": \"d73f11b3-9fb3-4c4f-aff6-3f02c7e294de\",\n      \"createTime\":
1388728576473,\n      \"lastUsedTime\": 1388728717138\n    },\n    {\n      \"id\": 17,\n
     \"clientInfo\": \"\",\n      \"host\": \"71F306F9-997C-40E1-8CCE-0611CF71B623\",\n  
   \"port\": 2179,\n      \"tag\": \"eaefccc1-6c41-4e57-9911-4dbacac7fe40\",\n      \"createTime\":
1388728570588,\n      \"lastUsedTime\": 1388728699119\n    },\n    {\n      \"id\": 20,\n
     \"clientInfo\": \"\",\n      \"host\": \"063CC300-DB0C-4CA3-986A-D8E80EC38031\",\n  
   \"port\": 2179,\n      \"tag\": \"af3bfdd9-2ca2-4dbd-ba99-e79f2a83319d\",\n      \"createTime\":
1388728604238,\n      \"lastUsedTime\": 1388728850905\n    }\n  ]\n}","wait":0}}] }
> 2014-01-03 11:31:05,869 DEBUG [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-3:null)
SeqA 4-10358: Sending Seq 4-10358:  { Ans: , MgmtId: 132129494109518, via: 4, Ver: v1, Flags:
100010, [{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}] }
> 2014-01-03 11:31:07,127 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-e32b0652)
Execute sync-queue item: SyncQueueItemVO {id:15, queueId: 13, contentType: AsyncJob, contentId:
48, lastProcessMsid: null, lastprocessNumber: null, lastProcessTime: null, created: Fri Jan
03 11:31:05 IST 2014}
> 2014-01-03 11:31:07,128 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-e32b0652)
Schedule queued job-48
> 2014-01-03 11:31:07,279 INFO  [o.a.c.f.j.i.AsyncJobMonitor] (Job-Executor-58:ctx-de8b3ed7)
Add job-48 into job monitoring
> 2014-01-03 11:31:07,279 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Job-Executor-58:ctx-de8b3ed7)
Executing AsyncJobVO {id:48, userId: 2, accountId: 2, instanceType: null, instanceId: null,
cmd: com.cloud.vm.VmWorkMigrate, cmdInfo: rO0ABXNyABpjb20uY2xvdWQudm0uVm1Xb3JrTWlncmF0ZRdxQXtPtzYqAgAGSgAJc3JjSG9zdElkTAAJY2x1c3RlcklkdAAQTGphdmEvbGFuZy9Mb25nO0wABmhvc3RJZHEAfgABTAAFcG9kSWRxAH4AAUwAB3N0b3JhZ2V0AA9MamF2YS91dGlsL01hcDtMAAZ6b25lSWRxAH4AAXhyABNjb20uY2xvdWQudm0uVm1Xb3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VySWRKAAR2bUlkTAALaGFuZGxlck5hbWV0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAACAAAAAAAAAAIAAAAAAAAABHQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAnNyAA5qYXZhLmxhbmcuTG9uZzuL5JDMjyPfAgABSgAFdmFsdWV4cgAQamF2YS5sYW5nLk51bWJlcoaslR0LlOCLAgAAeHAAAAAAAAAAAXEAfgAJcQB-AAlwcQB-AAk,
cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid:
132129494109518, completeMsid: null, lastUpdated: null, lastPolled: null, created: Fri Jan
03 11:31:05 IST 2014}
> 2014-01-03 11:31:07,279 DEBUG [c.c.v.VmWorkJobDispatcher] (Job-Executor-58:ctx-de8b3ed7)
Run VM work job: com.cloud.vm.VmWorkMigrate
> 2014-01-03 11:31:07,281 DEBUG [c.c.v.VmWorkJobHandlerProxy] (Job-Executor-58:ctx-de8b3ed7
ctx-0b17c466) Execute VM work job: com.cloud.vm.VmWorkMigrate{"zoneId":1,"podId":1,"clusterId":1,"hostId":1,"srcHostId":2,"userId":2,"accountId":2,"vmId":4,"handlerName":"VirtualMachineManagerImpl"}
> 2014-01-03 11:31:07,283 ERROR [c.c.v.VmWorkJobHandlerProxy] (Job-Executor-58:ctx-de8b3ed7
ctx-0b17c466) Invocation exception, caused by: java.lang.NullPointerException
> 2014-01-03 11:31:07,283 ERROR [c.c.v.VmWorkJobDispatcher] (Job-Executor-58:ctx-de8b3ed7
ctx-0b17c466) Unable to complete AsyncJobVO {id:48, userId: 2, accountId: 2, instanceType:
null, instanceId: null, cmd: com.cloud.vm.VmWorkMigrate, cmdInfo: rO0ABXNyABpjb20uY2xvdWQudm0uVm1Xb3JrTWlncmF0ZRdxQXtPtzYqAgAGSgAJc3JjSG9zdElkTAAJY2x1c3RlcklkdAAQTGphdmEvbGFuZy9Mb25nO0wABmhvc3RJZHEAfgABTAAFcG9kSWRxAH4AAUwAB3N0b3JhZ2V0AA9MamF2YS91dGlsL01hcDtMAAZ6b25lSWRxAH4AAXhyABNjb20uY2xvdWQudm0uVm1Xb3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VySWRKAAR2bUlkTAALaGFuZGxlck5hbWV0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAACAAAAAAAAAAIAAAAAAAAABHQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAnNyAA5qYXZhLmxhbmcuTG9uZzuL5JDMjyPfAgABSgAFdmFsdWV4cgAQamF2YS5sYW5nLk51bWJlcoaslR0LlOCLAgAAeHAAAAAAAAAAAXEAfgAJcQB-AAlwcQB-AAk,
cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid:
132129494109518, completeMsid: null, lastUpdated: null, lastPolled: null, created: Fri Jan
03 11:31:05 IST 2014}
> java.lang.NullPointerException
>         at com.cloud.vm.VmWorkMigrate.getDeployDestination(VmWorkMigrate.java:60)
>         at com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrate(VirtualMachineManagerImpl.java:4758)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:616)
>         at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
>         at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4855)
>         at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:99)
>         at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:522)
>         at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>         at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>         at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>         at java.lang.Thread.run(Thread.java:679)
> 2014-01-03 11:31:07,284 ERROR [o.a.c.f.j.i.AsyncJobManagerImpl] (Job-Executor-58:ctx-de8b3ed7)
Unexpected exception
> java.lang.NullPointerException
> 2014-01-03 11:31:07,284 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Job-Executor-58:ctx-de8b3ed7)
Complete async job-48, jobStatus: FAILED, resultCode: 530, result: null
> 2014-01-03 11:31:07,379 INFO  [o.a.c.f.j.i.AsyncJobMonitor] (Job-Executor-59:ctx-cfa27c50)
Add job-47 into job monitoring
> 2014-01-03 11:31:07,380 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Job-Executor-59:ctx-cfa27c50)
Executing AsyncJobVO {id:47, userId: 2, accountId: 2, instanceType: None, instanceId: null,
cmd: org.apache.cloudstack.api.command.admin.systemvm.MigrateSystemVMCmd, cmdInfo: {"response":"json","sessionkey":"48pxYt3vmMxm4yn7pwbEszyG7V4\u003d","virtualmachineid":"df65d272-b64f-4b5e-9def-e0b34aa481e6","cmdEventType":"VM.MIGRATE","hostid":"a138433d-0aa6-4711-9559-9738ea7e03fc","ctxUserId":"2","httpmethod":"GET","_":"1388728864662","ctxAccountId":"2","ctxStartEventId":"109"},
cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid:
132129494109518, completeMsid: null, lastUpdated: null, lastPolled: null, created: Fri Jan
03 11:31:05 IST 2014}
> 2014-01-03 11:31:07,512 ERROR [o.a.c.f.j.i.AsyncJobManagerImpl] (Job-Executor-59:ctx-cfa27c50)
Unable to find a wakeup dispatcher from the joined job: AsyncJobVO {id:47, userId: 2, accountId:
2, instanceType: None, instanceId: null, cmd: org.apache.cloudstack.api.command.admin.systemvm.MigrateSystemVMCmd,
cmdInfo: {"response":"json","sessionkey":"48pxYt3vmMxm4yn7pwbEszyG7V4\u003d","virtualmachineid":"df65d272-b64f-4b5e-9def-e0b34aa481e6","cmdEventType":"VM.MIGRATE","hostid":"a138433d-0aa6-4711-9559-9738ea7e03fc","ctxUserId":"2","httpmethod":"GET","_":"1388728864662","ctxAccountId":"2","ctxStartEventId":"109"},
cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid:
132129494109518, completeMsid: null, lastUpdated: null, lastPolled: null, created: Fri Jan
03 11:31:05 IST 2014}
> 2014-01-03 11:31:07,513 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Job-Executor-59:ctx-cfa27c50)
Done executing org.apache.cloudstack.api.command.admin.systemvm.MigrateSystemVMCmd for job-47
> 2014-01-03 11:31:07,578 INFO  [o.a.c.f.j.i.AsyncJobMonitor] (Job-Executor-59:ctx-cfa27c50)
Remove job-47 from job monitoring
> 2014-01-03 11:31:07,638 DEBUG [o.a.c.f.j.i.SyncQueueManagerImpl] (Job-Executor-58:ctx-de8b3ed7)
Sync queue (13) is currently empty
> 2014-01-03 11:31:07,639 INFO  [o.a.c.f.j.i.AsyncJobMonitor] (Job-Executor-58:ctx-de8b3ed7)
Remove job-48 from job monitoring
> 2014-01-03 11:31:07,688 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Job-Executor-57:ctx-f75b2d46)
Complete async job-47, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed
to migrate the system vm"}
> 2014-01-03 11:31:07,745 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Job-Executor-57:ctx-f75b2d46)
Done executing org.apache.cloudstack.api.command.admin.systemvm.MigrateSystemVMCmd for job-47
> 2014-01-03 11:31:07,778 INFO  [o.a.c.f.j.i.AsyncJobMonitor] (Job-Executor-57:ctx-f75b2d46)
Remove job-47 from job monitoring
> Also din't see events related to live migration on the agent side. Looks like live migration
command has not been passed to the agent.
>  



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Mime
View raw message