cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dave Garbus <davegar...@spiceworks.com>
Subject Re: Xenserver 6.5 live migration with local volumes
Date Tue, 24 May 2016 15:57:21 GMT
I updated the issue referenced in this email (https://issues.apache.org/jira/browse/CLOUDSTACK-9353)
to include CloudStack 4.8 as an affected version. If we could at least get someone assigned
to it, that’d be a good start.

Thanks,

- Dave

On May 24, 2016, at 9:42 AM, Ivan Derbenev <ivan.derbenev@tech-corps.com<mailto:ivan.derbenev@tech-corps.com>>
wrote:

Is there a way I can fix this on my installation, without recompiling jar files?

Regards,
IT engineer
Farheap, Russia
Ivan Derbenev

-----Original Message-----
From: Koushik Das [mailto:koushik.das@accelerite.com]
Sent: Tuesday, May 24, 2016 2:06 PM
To: users@cloudstack.apache.org<mailto:users@cloudstack.apache.org>
Subject: Re: Xenserver 6.5 live migration with local volumes

This is a product bug and needs to be fixed. There is some serialization issue that needs
fixing.

In the below code volumeToFiler is null as Map is not a serializable type. It needs to be
replaced with getVolumeToFilerAsList() and then the data needs to be read appropriately.

       final Map<VolumeTO, StorageFilerTO> volumeToFiler = command.getVolumeToFiler();

-Koushik

On 24/05/16, 3:23 PM, "Ivan Derbenev" <ivan.derbenev@tech-corps.com<mailto:ivan.derbenev@tech-corps.com>>
wrote:

Hello!
After the migration onto XS 6.5 (and installing all the updates) and CS
4.8 I can't migrate VMs with Local Storage

The issue is similar to this:
https://issues.apache.org/jira/browse/CLOUDSTACK-9353, but it doesn't have any solutions Any
ideas?

Errors:

2016-05-24 12:10:16,440 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
(DirectAgentCronJob-26:ctx-ab4250ff) (logid:4fa92c13) Done with process
of VM state report. host: 136592
2016-05-24 12:10:16,450 DEBUG [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Looking for network
named cloud-main
2016-05-24 12:10:16,466 DEBUG [c.c.h.x.r.XsLocalNetwork]
(DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Found a network called
cloud-main on host=10.7.130.103;
Network=9632020b-4d98-1bed-9ae8-477e0993cab0;
pif=3c7d1121-2bde-26cf-9478-18e51ffca75f
2016-05-24 12:10:16,477 DEBUG [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Creating VLAN 1007 on
host 10.7.130.103 on device eth0
2016-05-24 12:10:17,163 DEBUG [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) VLAN is created for
1007.  The uuid is ec053fee-0426-9665-150a-77399562196b
2016-05-24 12:10:17,170 WARN
[c.c.h.x.r.w.x.XenServer610MigrateWithStorageCommandWrapper] (DirectAgent-132:ctx-a97d60fa)
(logid:8748bffe) Catch Exception java.lang.NullPointerException. Storage motion failed due
to java.lang.NullPointerException java.lang.NullPointerException
      at com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:86)
      at com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:54)
      at com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
      at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677)
      at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
      at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
      at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
      at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
      at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
      at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:745)
2016-05-24 12:10:17,171 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-132:ctx-a97d60fa)
(logid:8748bffe) Seq 136589-479633360314957887: Response Received:
2016-05-24 12:10:17,171 DEBUG [c.c.a.t.Request]
(DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Seq
136589-479633360314957887: Processing:  { Ans: , MgmtId:
128437433941265, via:
136589(cloudstack-computing03-sar01.tech-corps.com<http://cloudstack-computing03-sar01.tech-corps.com>),
Ver: v1, Flags:
110,
[{"com.cloud.agent.api.MigrateWithStorageAnswer":{"result":false,"detai
ls":"Exception: java.lang.NullPointerException\nMessage: null\nStack:
java.lang.NullPointerException\n\tat
com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610Migr
ateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageComm
andWrapper.java:86)\n\tat
com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610Migr
ateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageComm
andWrapper.java:54)\n\tat
com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWr
apper.execute(CitrixRequestWrapper.java:122)\n\tat
com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeReque
st(CitrixResourceBase.java:1677)\n\tat
com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgen
tAttache.java:315)\n\tat
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(Mana
gedContextRunnable.java:49)\n\tat
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call
(DefaultManagedContext.java:56)\n\tat
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWi
thContext(DefaultManagedContext.java:103)\n\tat
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWit
hContext(DefaultManagedContext.java:53)\n\tat
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(Manage
dContextRunnable.java:46)\n\tat
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:262)\n\tat
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.ac
cess$201(ScheduledThreadPoolExecutor.java:178)\n\tat
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.ru<http://ScheduledFutureTask.ru>
n(ScheduledThreadPoolExecutor.java:292)\n\tat
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.ja
va:1145)\n\tat
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.j
ava:615)\n\tat java.lang.Thread.run(Thread.java:745)\n","wait":0}}] }
2016-05-24 12:10:17,171 DEBUG [c.c.a.m.AgentAttache]
(DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Seq
136589-479633360314957887: No more commands found
2016-05-24 12:10:17,171 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
(logid:8748bffe) Seq 136589-479633360314957887: Received:  { Ans: ,
MgmtId: 128437433941265, via:
136589(cloudstack-computing03-sar01.tech-corps.com<http://cloudstack-computing03-sar01.tech-corps.com>),
Ver: v1, Flags:
110, { MigrateWithStorageAnswer } }
2016-05-24 12:10:17,171 ERROR
[o.a.c.s.m.XenServerStorageMotionStrategy]
(Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
(logid:8748bffe) Migration with storage of vm VM[User|i-3-271-VM]
failed. Details: Exception: java.lang.NullPointerException
Message: null
Stack: java.lang.NullPointerException
      at com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:86)
      at com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:54)
      at com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
      at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677)
      at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
      at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
      at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
      at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
      at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
      at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:745)

2016-05-24 12:10:17,171 ERROR
[o.a.c.s.m.XenServerStorageMotionStrategy]
(Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
(logid:8748bffe) copy failed
com.cloud.utils.exception.CloudRuntimeException: Error while migrating
the vm VM[User|i-3-271-VM] to host Host[-136589-Routing]. Exception:
java.lang.NullPointerException
Message: null
Stack: java.lang.NullPointerException
      at com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:86)
      at com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:54)
      at com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
      at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677)
      at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
      at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
      at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
      at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
      at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
      at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:745)

      at org.apache.cloudstack.storage.motion.XenServerStorageMotionStrategy.migrateVmWithVolumesWithinCluster(XenServerStorageMotionStrategy.java:212)
      at org.apache.cloudstack.storage.motion.XenServerStorageMotionStrategy.copyAsync(XenServerStorageMotionStrategy.java:116)
      at org.apache.cloudstack.storage.motion.DataMotionServiceImpl.copyAsync(DataMotionServiceImpl.java:90)
      at org.apache.cloudstack.storage.volume.VolumeServiceImpl.migrateVolumes(VolumeServiceImpl.java:1160)
      at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolumes(VolumeOrchestrator.java:1002)
      at com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:2274)
      at com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:4638)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:606)
      at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
      at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4741)
      at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
      at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554)
      at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
      at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
      at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
      at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
      at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
      at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:745)
2016-05-24 12:10:17,183 WARN  [o.a.c.s.d.ObjectInDataStoreManagerImpl]
(Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
(logid:8748bffe) Unsupported data object (VOLUME,
org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@142a92a2),
no need to delete from object in store ref table
2016-05-24 12:10:17,183 DEBUG [o.a.c.e.o.VolumeOrchestrator]
(Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
(logid:8748bffe) Failed to migrated vm VM[User|i-3-271-VM] along with
its volumes. com.cloud.utils.exception.CloudRuntimeException: Error
while migrating the vm VM[User|i-3-271-VM] to host
Host[-136589-Routing]. Exception: java.lang.NullPointerException
Message: null
Stack: java.lang.NullPointerException
      at com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:86)
      at com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:54)
      at com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
      at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677)
      at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
      at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
      at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
      at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
      at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
      at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:745)

2016-05-24 12:10:17,183 INFO  [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
(logid:8748bffe) Migration was unsuccessful.  Cleaning up:
VM[User|i-3-271-VM]
2016-05-24 12:10:17,184 WARN  [o.a.c.alerts]
(Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
(logid:8748bffe)  alertType:: 17 // dataCenterId:: 1 // podId:: 1 //
clusterId:: null // message:: Unable to migrate vm i-3-271-VM from host
cloudstack-computing01-sar01.tech-corps.com<http://cloudstack-computing01-sar01.tech-corps.com>
in zone zone1.sar and pod
zone1.sar
2016-05-24 12:10:17,185 DEBUG [c.c.a.AlertManagerImpl]
(Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
(logid:8748bffe) Have already sent: 1 emails for alert type '17' --
skipping send email
2016-05-24 12:10:17,185 DEBUG [c.c.a.m.AgentAttache]
(Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
(logid:8748bffe) Seq 136589-479633360314957888: Routed from
128437433941265
2016-05-24 12:10:17,185 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
(logid:8748bffe) Seq 136589-479633360314957888: Sending  { Cmd ,
MgmtId: 128437433941265, via:
136589(cloudstack-computing03-sar01.tech-corps.com<http://cloudstack-computing03-sar01.tech-corps.com>),
Ver: v1, Flags:
100011,
[{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence
":false,"checkBeforeCleanup":false,"vmName":"i-3-271-VM","wait":0}}] }
2016-05-24 12:10:17,185 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
(logid:8748bffe) Seq 136589-479633360314957888: Executing:  { Cmd ,
MgmtId: 128437433941265, via:
136589(cloudstack-computing03-sar01.tech-corps.com<http://cloudstack-computing03-sar01.tech-corps.com>),
Ver: v1, Flags:
100011,
[{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence
":false,"checkBeforeCleanup":false,"vmName":"i-3-271-VM","wait":0}}] }
2016-05-24 12:10:17,185 DEBUG [c.c.a.m.DirectAgentAttache]
(DirectAgent-34:ctx-65ef585a) (logid:924b5b07) Seq
136589-479633360314957888: Executing request
2016-05-24 12:10:17,197 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
(logid:8748bffe) VM state transitted from :Migrating to Running with
event: OperationFailedvm's original host id: 136592 new host id: 136592
host id before state transition: 136589
2016-05-24 12:10:17,200 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
(logid:8748bffe) Hosts's actual total CPU: 15960 and CPU after applying
overprovisioning: 39900
2016-05-24 12:10:17,200 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
(logid:8748bffe) Hosts's actual total RAM: 11549697280 and RAM after
applying overprovisioning: 11549697024
2016-05-24 12:10:17,200 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
(logid:8748bffe) release cpu from host: 136589, old used:
3990,reserved: 0, actual total: 15960, total with overprovisioning:
39900; new used: 1995,reserved:0; movedfromreserved:
false,moveToReserveredfalse
2016-05-24 12:10:17,200 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
(logid:8748bffe) release mem from host: 136589, old used:
5368709120,reserved: 0, total: 11549697024; new used:
4294967296,reserved:0; movedfromreserved: false,moveToReserveredfalse
2016-05-24 12:10:17,201 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-34:ctx-65ef585a) (logid:8748bffe)
Seq 136589-479633360314957888: Response Received:
2016-05-24 12:10:17,202 DEBUG [c.c.a.t.Request]
(DirectAgent-34:ctx-65ef585a) (logid:8748bffe) Seq
136589-479633360314957888: Processing:  { Ans: , MgmtId:
128437433941265, via:
136589(cloudstack-computing03-sar01.tech-corps.com<http://cloudstack-computing03-sar01.tech-corps.com>),
Ver: v1, Flags:
10, [{"com.cloud.agent.api.StopAnswer":{"result":true,"details":"VM
does not exist","wait":0}}] }
2016-05-24 12:10:17,202 DEBUG [c.c.a.m.AgentAttache] (DirectAgent-34:ctx-65ef585a) (logid:8748bffe)
Seq 136589-479633360314957888: Unable to find listener.
2016-05-24 12:10:17,218 ERROR [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-3:ctx-601b1307
job-4874/job-4875 ctx-619c9d4e) (logid:8748bffe) Invocation exception, caused by: com.cloud.utils.exception.CloudRuntimeException:
Failed to migrated vm VM[User|i-3-271-VM] along with its volumes.
2016-05-24 12:10:17,218 INFO  [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-3:ctx-601b1307
job-4874/job-4875 ctx-619c9d4e) (logid:8748bffe) Rethrow exception com.cloud.utils.exception.CloudRuntimeException:
Failed to migrated vm VM[User|i-3-271-VM] along with its volumes.
2016-05-24 12:10:17,218 DEBUG [c.c.v.VmWorkJobDispatcher]
(Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
Done with run of VM work job: com.cloud.vm.VmWorkMigrateWithStorage for
VM 271, job origin: 4874
2016-05-24 12:10:17,218 ERROR [c.c.v.VmWorkJobDispatcher]
(Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
Unable to complete AsyncJobVO {id:4875, userId: 17, accountId: 3,
instanceType: null, instanceId: null, cmd:
com.cloud.vm.VmWorkMigrateWithStorage, cmdInfo:
rO0ABXNyACVjb20uY2xvdWQudm0uVm1Xb3JrTWlncmF0ZVdpdGhTdG9yYWdlsew9z6UxtXM
CAANKAApkZXN0SG9zdElkSgAJc3JjSG9zdElkTAAMdm9sdW1lVG9Qb29sdAAPTGphdmEvdX
RpbC9NYXA7eHIAE2NvbS5jbG91ZC52bS5WbVdvcmufmbZW8CVnawIABEoACWFjY291bnRJZ
EoABnVzZXJJZEoABHZtSWRMAAtoYW5kbGVyTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO3hw
AAAAAAAAAAMAAAAAAAAAEQAAAAAAAAEPdAAZVmlydHVhbE1hY2hpbmVNYW5hZ2VySW1wbAA
AAAAAAhWNAAAAAAACFZBzcgARamF2YS51dGlsLkhhc2hNYXAFB9rBwxZg0QMAAkYACmxvYW
RGYWN0b3JJAAl0aHJlc2hvbGR4cD9AAAAAAAAQdwgAAAAQAAAAAHg, cmdVersion: 0,
status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null,
initMsid: 128437433941265, completeMsid: null, lastUpdated: null,
lastPolled: null, created: Tue May 24 12:10:16 MSK 2016}, job
origin:4874
com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm VM[User|i-3-271-VM]
along with its volumes.
      at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolumes(VolumeOrchestrator.java:1007)
      at com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:2274)
      at com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:4638)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:606)
      at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
      at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4741)
      at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
      at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554)
      at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
      at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
      at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
      at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
      at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
      at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:745)
2016-05-24 12:10:17,220 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
Complete async job-4875, jobStatus: FAILED, resultCode: 0, result:
rO0ABXNyAC9jb20uY2xvdWQudXRpbHMuZXhjZXB0aW9uLkNsb3VkUnVudGltZUV4Y2VwdGl
vbgAAAABWTU9yAwABSQALY3NFcnJvckNvZGV4cgAaamF2YS5sYW5nLlJ1bnRpbWVFeGNlcH
Rpb26eXwZHCjSD5QIAAHhyABNqYXZhLmxhbmcuRXhjZXB0aW9u0P0fPho7HMQCAAB4cgATa
mF2YS5sYW5nLlRocm93YWJsZdXGNSc5d7jLAwAETAAFY2F1c2V0ABVMamF2YS9sYW5nL1Ro
cm93YWJsZTtMAA1kZXRhaWxNZXNzYWdldAASTGphdmEvbGFuZy9TdHJpbmc7WwAKc3RhY2t
UcmFjZXQAHltMamF2YS9sYW5nL1N0YWNrVHJhY2VFbGVtZW50O0wAFHN1cHByZXNzZWRFeG
NlcHRpb25zdAAQTGphdmEvdXRpbC9MaXN0O3hwcQB-AAh0AEJGYWlsZWQgdG8gbWlncmF0Z
WQgdm0gVk1bVXNlcnxpLTMtMjcxLVZNXSBhbG9uZyB3aXRoIGl0cyB2b2x1bWVzLiB1cgAe
W0xqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnQ7AkYqPDz9IjkCAAB4cAAAABZzcgAbamF
2YS5sYW5nLlN0YWNrVHJhY2VFbGVtZW50YQnFmiY23YUCAARJAApsaW5lTnVtYmVyTAAOZG
VjbGFyaW5nQ2xhc3NxAH4ABUwACGZpbGVOYW1lcQB-AAVMAAptZXRob2ROYW1lcQB-AAV4c
AAAA-90AD1vcmcuYXBhY2hlLmNsb3Vkc3RhY2suZW5naW5lLm9yY2hlc3RyYXRpb24uVm9s
dW1lT3JjaGVzdHJhdG9ydAAXVm9sdW1lT3JjaGVzdHJhdG9yLmphdmF0AA5taWdyYXRlVm9
sdW1lc3NxAH4ADAAACOJ0ACZjb20uY2xvdWQudm0uVmlydHVhbE1hY2hpbmVNYW5hZ2VySW
1wbHQAHlZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwuamF2YXQAHW9yY2hlc3RyYXRlTWlnc
mF0ZVdpdGhTdG9yYWdlc3EAfgAMAAASHnEAfgAScQB-ABNxAH4AFHNxAH4ADP____50ACRz
dW4ucmVmbGVjdC5OYXRpdmVNZXRob2RBY2Nlc3NvckltcGx0AB1OYXRpdmVNZXRob2RBY2N
lc3NvckltcGwuamF2YXQAB2ludm9rZTBzcQB-AAwAAAA5cQB-ABdxAH4AGHQABmludm9rZX
NxAH4ADAAAACt0AChzdW4ucmVmbGVjdC5EZWxlZ2F0aW5nTWV0aG9kQWNjZXNzb3JJbXBsd
AAhRGVsZWdhdGluZ01ldGhvZEFjY2Vzc29ySW1wbC5qYXZhcQB-ABtzcQB-AAwAAAJedAAY
amF2YS5sYW5nLnJlZmxlY3QuTWV0aG9kdAALTWV0aG9kLmphdmFxAH4AG3NxAH4ADAAAAGt
0ACJjb20uY2xvdWQudm0uVm1Xb3JrSm9iSGFuZGxlclByb3h5dAAaVm1Xb3JrSm9iSGFuZG
xlclByb3h5LmphdmF0AA9oYW5kbGVWbVdvcmtKb2JzcQB-AAwAABKFcQB-ABJxAH4AE3EAf
gAlc3EAfgAMAAAAZnQAIGNvbS5jbG91ZC52bS5WbVdvcmtKb2JEaXNwYXRjaGVydAAYVm1X
b3JrSm9iRGlzcGF0Y2hlci5qYXZhdAAGcnVuSm9ic3EAfgAMAAACKnQAP29yZy5hcGFjaGU
uY2xvdWRzdGFjay5mcmFtZXdvcmsuam9icy5pbXBsLkFzeW5jSm9iTWFuYWdlckltcGwkNX
QAGEFzeW5jSm9iTWFuYWdlckltcGwuamF2YXQADHJ1bkluQ29udGV4dHNxAH4ADAAAADF0A
D5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0
UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUuamF2YXQAA3J1bnNxAH4ADAA
AADh0AEJvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0LmltcGwuRGVmYX
VsdE1hbmFnZWRDb250ZXh0JDF0ABpEZWZhdWx0TWFuYWdlZENvbnRleHQuamF2YXQABGNhb
GxzcQB-AAwAAABndABAb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5p
bXBsLkRlZmF1bHRNYW5hZ2VkQ29udGV4dHEAfgA1dAAPY2FsbFdpdGhDb250ZXh0c3EAfgA
MAAAANXEAfgA4cQB-ADV0AA5ydW5XaXRoQ29udGV4dHNxAH4ADAAAAC50ADxvcmcuYXBhY2
hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGVxA
H4AMXEAfgAyc3EAfgAMAAAB9nEAfgAscQB-AC1xAH4AMnNxAH4ADAAAAdd0AC5qYXZhLnV0
aWwuY29uY3VycmVudC5FeGVjdXRvcnMkUnVubmFibGVBZGFwdGVydAAORXhlY3V0b3JzLmp
hdmFxAH4ANnNxAH4ADAAAAQZ0AB9qYXZhLnV0aWwuY29uY3VycmVudC5GdXR1cmVUYXNrdA
APRnV0dXJlVGFzay5qYXZhcQB-ADJzcQB-AAwAAAR5dAAnamF2YS51dGlsLmNvbmN1cnJlb
nQuVGhyZWFkUG9vbEV4ZWN1dG9ydAAXVGhyZWFkUG9vbEV4ZWN1dG9yLmphdmF0AAlydW5X
b3JrZXJzcQB-AAwAAAJndAAuamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN
1dG9yJFdvcmtlcnEAfgBHcQB-ADJzcQB-AAwAAALpdAAQamF2YS5sYW5nLlRocmVhZHQAC1
RocmVhZC5qYXZhcQB-ADJzcgAmamF2YS51dGlsLkNvbGxlY3Rpb25zJFVubW9kaWZpYWJsZ
Uxpc3T8DyUxteyOEAIAAUwABGxpc3RxAH4AB3hyACxqYXZhLnV0aWwuQ29sbGVjdGlvbnMk
VW5tb2RpZmlhYmxlQ29sbGVjdGlvbhlCAIDLXvceAgABTAABY3QAFkxqYXZhL3V0aWwvQ29
sbGVjdGlvbjt4cHNyABNqYXZhLnV0aWwuQXJyYXlMaXN0eIHSHZnHYZ0DAAFJAARzaXpleH
AAAAAAdwQAAAAAeHEAfgBTeAAAEJp3CAAAAAAAAAAAeA
2016-05-24 12:10:17,220 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
Publish async job-4875 complete on message bus
2016-05-24 12:10:17,220 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
Wake up jobs related to job-4875
2016-05-24 12:10:17,220 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
Update db status for job-4875
2016-05-24 12:10:17,221 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
Wake up jobs joined with job-4875 and disjoin all subjobs created from
job- 4875
2016-05-24 12:10:17,230 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
Done executing com.cloud.vm.VmWorkMigrateWithStorage for job-4875
2016-05-24 12:10:17,231 INFO  [o.a.c.f.j.i.AsyncJobMonitor]
(Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
Remove job-4875 from job monitoring
2016-05-24 12:10:17,252 ERROR [c.c.a.ApiAsyncJobDispatcher]
(API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Unexpected
exception while executing
org.apache.cloudstack.api.command.admin.vm.MigrateVirtualMachineWithVol
umeCmd
com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm VM[User|i-3-271-VM]
along with its volumes.
      at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolumes(VolumeOrchestrator.java:1007)
      at com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:2274)
      at com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:4638)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:606)
      at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
      at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4741)
      at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
      at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554)
      at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
      at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
      at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
      at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
      at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
      at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:745)
2016-05-24 12:10:17,253 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Complete
async job-4874, jobStatus: FAILED, resultCode: 530, result:
org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[
],"errorcode":530,"errortext":"Failed to migrated vm
VM[User|i-3-271-VM] along with its volumes. "}
2016-05-24 12:10:17,254 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Publish
async job-4874 complete on message bus
2016-05-24 12:10:17,254 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Wake up
jobs related to job-4874
2016-05-24 12:10:17,254 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Update db
status for job-4874
2016-05-24 12:10:17,255 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Wake up
jobs joined with job-4874 and disjoin all subjobs created from job-
4874
2016-05-24 12:10:17,263 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Done
executing
org.apache.cloudstack.api.command.admin.vm.MigrateVirtualMachineWithVol
umeCmd for job-4874
2016-05-24 12:10:17,263 INFO  [o.a.c.f.j.i.AsyncJobMonitor]
(API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Remove
job-4874 from job monitoring ^C


Regards,
IT engineer
Farheap, Russia
Ivan Derbenev





DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Accelerite,
a Persistent Systems business. It is intended only for the use of the individual or entity
to which it is addressed. If you are not the intended recipient, you are not authorized to
read, retain, copy, print, distribute or use this message. If you have received this communication
in error, please notify the sender and delete all copies of this message. Accelerite, a Persistent
Systems business does not accept any liability for virus infected mails.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message