cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marty Godsey <ma...@gonsource.com>
Subject RE: Unable to start a VM due to insufficient capacity
Date Fri, 21 Apr 2017 23:21:48 GMT
Without looking too deep into this it seems that there maybe a stuck task trying to snapshot
the VHD.

result:
            errorInfo: [OTHER_OPERATION_IN_PROGRESS, VDI, OpaqueRef:1d681250-d950-81ec-9f87-f7eb3d8b0cc1,
snapshot]
          otherConfig: {}

Again, based on what you sent, I would look at any hung xapi tasks on the host in question.

Regards,
Marty Godsey
nSource Solutions

-----Original Message-----
From: J Andersons 
Sent: Friday, April 21, 2017 2:20 AM
To: users@cloudstack.apache.org
Subject: Unable to start a VM due to insufficient capacity

Hi!
Just like that it is impossible to start any vm. ACS4.8, Xenservers 6.5 Unable to start a
VM due to insufficient capacity. I can see in xencenter it is trying to start VM on all hosts
but fails. Also there is enough resources on hosts.
Maybe its an XenServer issue, because it is also unable to stop VMS.

In log when starting VM:
2017-04-21 08:50:35,806 WARN  [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-28:ctx-6bbd1566) (logid:91c36627) Task failed! Task 
record:                 uuid: a9ac4c12-a6ba-6d9a-556f-b5d88f8cd2d3
            nameLabel: Async.VM.start_on
      nameDescription:
    allowedOperations: []
    currentOperations: {}
              created: Fri Apr 21 08:48:34 EEST 2017
             finished: Fri Apr 21 08:50:35 EEST 2017
               status: failure
           residentOn: com.xensource.xenapi.Host@c07df7eb
             progress: 1.0
                 type: <none/>
               result:
            errorInfo: [OTHER_OPERATION_IN_PROGRESS, VDI, OpaqueRef:1d681250-d950-81ec-9f87-f7eb3d8b0cc1,
snapshot]
          otherConfig: {}
            subtaskOf: com.xensource.xenapi.Task@aaf13f6f
             subtasks: []

2017-04-21 08:50:35,818 WARN  [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-28:ctx-6bbd1566) (logid:91c36627) Unable to start
VM(i-95-292-VM) on host(a6601054-b730-460a-b654-a29f1a5b8a1c) due to 
Task failed! Task record:                 uuid: 
a9ac4c12-a6ba-6d9a-556f-b5d88f8cd2d3
            nameLabel: Async.VM.start_on
      nameDescription:
    allowedOperations: []
    currentOperations: {}
              created: Fri Apr 21 08:48:34 EEST 2017
             finished: Fri Apr 21 08:50:35 EEST 2017
               status: failure
           residentOn: com.xensource.xenapi.Host@c07df7eb
             progress: 1.0
                 type: <none/>
               result:
            errorInfo: [OTHER_OPERATION_IN_PROGRESS, VDI, OpaqueRef:1d681250-d950-81ec-9f87-f7eb3d8b0cc1,
snapshot]
          otherConfig: {}
            subtaskOf: com.xensource.xenapi.Task@aaf13f6f
             subtasks: []

Task failed! Task record:                 uuid: 
a9ac4c12-a6ba-6d9a-556f-b5d88f8cd2d3
            nameLabel: Async.VM.start_on
      nameDescription:
    allowedOperations: []
    currentOperations: {}
              created: Fri Apr 21 08:48:34 EEST 2017
             finished: Fri Apr 21 08:50:35 EEST 2017
               status: failure
           residentOn: com.xensource.xenapi.Host@c07df7eb
             progress: 1.0
                 type: <none/>
               result:
            errorInfo: [OTHER_OPERATION_IN_PROGRESS, VDI, OpaqueRef:1d681250-d950-81ec-9f87-f7eb3d8b0cc1,
snapshot]
          otherConfig: {}
            subtaskOf: com.xensource.xenapi.Task@aaf13f6f
             subtasks: []

         at
com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.checkForSuccess(CitrixResourceBase.java:461)
         at
com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.startVM(CitrixResourceBase.java:4807)
         at
com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStartCommandWrapper.execute(CitrixStartCommandWrapper.java:126)
         at
com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStartCommandWrapper.execute(CitrixStartCommandWrapper.java:53)
         at
com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
         at
com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677)
         at
com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
         at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
         at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
         at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
         at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
         at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
         at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
         at java.util.concurrent.FutureTask.run(FutureTask.java:262)
         at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
         at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
         at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
         at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
         at java.lang.Thread.run(Thread.java:745)
2017-04-21 08:50:35,826 DEBUG [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-28:ctx-6bbd1566) (logid:91c36627) unable to destroy
task(com.xensource.xenapi.Task@27d940e) on
host(a6601054-b730-460a-b654-a29f1a5b8a1c) due to You gave an invalid object reference.  The
object may have recently been deleted.  The class parameter gives the type of reference given,
and the handle parameter echoes the bad value given.
2017-04-21 08:50:35,826 WARN [c.c.h.x.r.w.x.CitrixStartCommandWrapper]
(DirectAgent-28:ctx-6bbd1566) (logid:91c36627) Catch Exception: class com.cloud.utils.exception.CloudRuntimeException
due to
com.cloud.utils.exception.CloudRuntimeException: Unable to start
VM(i-95-292-VM) on host(a6601054-b730-460a-b654-a29f1a5b8a1c) due to 
Task failed! Task record:                 uuid: 
a9ac4c12-a6ba-6d9a-556f-b5d88f8cd2d3
            nameLabel: Async.VM.start_on
      nameDescription:
    allowedOperations: []
    currentOperations: {}
              created: Fri Apr 21 08:48:34 EEST 2017
             finished: Fri Apr 21 08:50:35 EEST 2017
               status: failure
           residentOn: com.xensource.xenapi.Host@c07df7eb
             progress: 1.0
                 type: <none/>
               result:
            errorInfo: [OTHER_OPERATION_IN_PROGRESS, VDI, OpaqueRef:1d681250-d950-81ec-9f87-f7eb3d8b0cc1,
snapshot]
          otherConfig: {}
            subtaskOf: com.xensource.xenapi.Task@aaf13f6f
             subtasks: []

com.cloud.utils.exception.CloudRuntimeException: Unable to start
VM(i-95-292-VM) on host(a6601054-b730-460a-b654-a29f1a5b8a1c) due to 
Task failed! Task record:                 uuid: 
a9ac4c12-a6ba-6d9a-556f-b5d88f8cd2d3
            nameLabel: Async.VM.start_on
      nameDescription:
    allowedOperations: []
    currentOperations: {}
              created: Fri Apr 21 08:48:34 EEST 2017
             finished: Fri Apr 21 08:50:35 EEST 2017
               status: failure
           residentOn: com.xensource.xenapi.Host@c07df7eb
             progress: 1.0
                 type: <none/>
               result:
            errorInfo: [OTHER_OPERATION_IN_PROGRESS, VDI, OpaqueRef:1d681250-d950-81ec-9f87-f7eb3d8b0cc1,
snapshot]
          otherConfig: {}
            subtaskOf: com.xensource.xenapi.Task@aaf13f6f
             subtasks: []

         at
com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.startVM(CitrixResourceBase.java:4826)
         at
com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStartCommandWrapper.execute(CitrixStartCommandWrapper.java:126)
         at
com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStartCommandWrapper.execute(CitrixStartCommandWrapper.java:53)
         at
com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
         at
com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677)
         at
com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
         at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
         at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
         at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
         at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
         at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
         at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
         at java.util.concurrent.FutureTask.run(FutureTask.java:262)
         at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
         at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
         at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
         at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
         at java.lang.Thread.run(Thread.java:745)
2017-04-21 08:50:35,828 WARN  [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-28:ctx-6bbd1566) (logid:91c36627) Unable to start i-95-292-VM due to
com.cloud.utils.exception.CloudRuntimeException: Unable to start
VM(i-95-292-VM) on host(a6601054-b730-460a-b654-a29f1a5b8a1c) due to 
Task failed! Task record:                 uuid: 
a9ac4c12-a6ba-6d9a-556f-b5d88f8cd2d3
            nameLabel: Async.VM.start_on
      nameDescription:
    allowedOperations: []
    currentOperations: {}
              created: Fri Apr 21 08:48:34 EEST 2017
             finished: Fri Apr 21 08:50:35 EEST 2017
               status: failure
           residentOn: com.xensource.xenapi.Host@c07df7eb
             progress: 1.0
                 type: <none/>
               result:
            errorInfo: [OTHER_OPERATION_IN_PROGRESS, VDI, OpaqueRef:1d681250-d950-81ec-9f87-f7eb3d8b0cc1,
snapshot]
          otherConfig: {}
            subtaskOf: com.xensource.xenapi.Task@aaf13f6f
             subtasks: []

         at
com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.startVM(CitrixResourceBase.java:4826)
         at
com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStartCommandWrapper.execute(CitrixStartCommandWrapper.java:126)
         at
com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStartCommandWrapper.execute(CitrixStartCommandWrapper.java:53)
         at
com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
         at
com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677)
         at
com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
         at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
         at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
         at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
         at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
         at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
         at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
         at java.util.concurrent.FutureTask.run(FutureTask.java:262)
         at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
         at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
         at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
         at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
         at java.lang.Thread.run(Thread.java:745)
2017-04-21 08:50:35,889 WARN  [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-28:ctx-6bbd1566) (logid:91c36627) Unable to clean up VBD due to You gave an invalid
object reference.  The object may have recently been deleted.  The class parameter gives the
type of reference given, and the handle parameter echoes the bad value given.
         at com.xensource.xenapi.Types.checkResponse(Types.java:693)
         at com.xensource.xenapi.Connection.dispatch(Connection.java:395)
         at
com.cloud.hypervisor.xenserver.resource.XenServerConnectionPool$XenServerConnection.dispatch(XenServerConnectionPool.java:457)
         at com.xensource.xenapi.VBD.unplug(VBD.java:1109)
         at
com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.handleVmStartFailure(CitrixResourceBase.java:3445)
         at
com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStartCommandWrapper.execute(CitrixStartCommandWrapper.java:208)
         at
com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStartCommandWrapper.execute(CitrixStartCommandWrapper.java:53)
         at
com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
         at
com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677)
         at
com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
         at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
         at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
         at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
         at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
         at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
         at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
         at java.util.concurrent.FutureTask.run(FutureTask.java:262)
         at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
         at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
         at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
         at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
         at java.lang.Thread.run(Thread.java:745)
2017-04-21 08:50:35,901 WARN  [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-28:ctx-6bbd1566) (logid:91c36627) Unable to clean up VBD due to You gave an invalid
object reference.  The object may have recently been deleted.  The class parameter gives the
type of reference given, and the handle parameter echoes the bad value given.
         at com.xensource.xenapi.Types.checkResponse(Types.java:693)
         at com.xensource.xenapi.Connection.dispatch(Connection.java:395)
         at
com.cloud.hypervisor.xenserver.resource.XenServerConnectionPool$XenServerConnection.dispatch(XenServerConnectionPool.java:457)
         at com.xensource.xenapi.VBD.unplug(VBD.java:1109)
         at
com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.handleVmStartFailure(CitrixResourceBase.java:3445)
         at
com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStartCommandWrapper.execute(CitrixStartCommandWrapper.java:208)
         at
com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStartCommandWrapper.execute(CitrixStartCommandWrapper.java:53)
         at
com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
         at
com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677)
         at
com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
         at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
         at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
         at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
         at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
         at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
         at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
         at java.util.concurrent.FutureTask.run(FutureTask.java:262)
         at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
         at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
         at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
         at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
         at java.lang.Thread.run(Thread.java:745)
2017-04-21 08:50:35,913 WARN  [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-28:ctx-6bbd1566) (logid:91c36627) Unable to cleanup VIF You gave an invalid object
reference.  The object may have recently been deleted.  The class parameter gives the type
of reference given, and the handle parameter echoes the bad value given.
         at com.xensource.xenapi.Types.checkResponse(Types.java:693)
         at com.xensource.xenapi.Connection.dispatch(Connection.java:395)
         at
com.cloud.hypervisor.xenserver.resource.XenServerConnectionPool$XenServerConnection.dispatch(XenServerConnectionPool.java:457)
         at com.xensource.xenapi.VIF.unplug(VIF.java:921)
         at
com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.handleVmStartFailure(CitrixResourceBase.java:3453)
         at
com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStartCommandWrapper.execute(CitrixStartCommandWrapper.java:208)
         at
com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStartCommandWrapper.execute(CitrixStartCommandWrapper.java:53)
         at
com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
         at
com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677)
         at
com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
         at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
         at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
         at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
         at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
         at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
         at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
         at java.util.concurrent.FutureTask.run(FutureTask.java:262)
         at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
         at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
         at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
         at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
         at java.lang.Thread.run(Thread.java:745)
2017-04-21 08:50:36,091 DEBUG [c.c.h.x.r.w.x.CitrixStartCommandWrapper]
(DirectAgent-28:ctx-6bbd1566) (logid:91c36627) The VM is in stopped state, detected problem
during startup : i-95-292-VM


--
J Andersons

Mime
View raw message