Mailing-List: contact issues-help@cloudstack.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cloudstack.apache.org
Date: Mon, 6 Jan 2014 12:57:51 +0000 (UTC)
From: "ASF subversion and git services (JIRA)" <jira@apache.org>
To: cloudstack-issues@incubator.apache.org
Message-ID: <JIRA.12676591.1383127511070.46806.1389013071634@arcas>
In-Reply-To: <JIRA.12676591.1383127511070@arcas>
References: <JIRA.12676591.1383127511070@arcas>
Subject: [jira] [Commented] (CLOUDSTACK-5002) unable to destroy vm ;VM
 destroy failed in Stop i-2-59-VM Command due to You gave an invalid object
 reference.  The object may have recently been deleted.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/CLOUDSTACK-5002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13862960#comment-13862960 ] 

ASF subversion and git services commented on CLOUDSTACK-5002:
-------------------------------------------------------------

Commit 2b2d5998d7735d8c01ba015629e56a2034673c87 in branch refs/heads/4.3 from [~koushikd]
[ https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;h=2b2d599 ]

CLOUDSTACK-5002: unable to destroy vm ;VM destroy failed in Stop i-2-59-VM Command due to You gave an invalid object reference. The object may have recently been deleted.

This is happening as concurrent operations are happening on the same VM. Earlier this was not seen as all vm operations were synchronized at agent layer. By making execute.in.sequence
global config to false this restriction is no longer there. In the latest code operations to a single vm are synchronized by maintaining a job queue. In some scenarios the destroy vm operation
was not going through this job queue mechanism and so was resulting in failures due to simultaneous operations.


> unable to destroy vm ;VM destroy failed in Stop i-2-59-VM Command due to You gave an invalid object reference.  The object may have recently been deleted. 
> -----------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: CLOUDSTACK-5002
>                 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-5002
>             Project: CloudStack
>          Issue Type: Bug
>      Security Level: Public(Anyone can view this level - this is the default.) 
>          Components: Management Server, XenServer
>    Affects Versions: 4.2.1
>            Reporter: prashant kumar mishra
>            Assignee: Koushik Das
>            Priority: Critical
>             Fix For: 4.3.0
>
>         Attachments: MS_xen_DB.rar, SMlog
>
>
> steps to reproduce
> ---------------------------
> 1-prepare CS setup with xen 6.1
> 2-set execute.in.sequence.hypervisor.commands and execute.in.sequence.network.element.commands to false
> 3-Restart MS
> 4-deploy 35 vms parallelly with default centOS template
> 5-try to destroy all uservms using script 
> Expected
> -------------
> All vms should get destroyed
> Actual
> ------------
> 7 vms went in stopped state
> Logs
> -------
> 2013-10-30 10:22:35,406 DEBUG [agent.transport.Request] (Job-Executor-93:job-244 = [ 539bf83c-67bb-4d4a-a8e2-837a8d85cb4d ]) Seq 3-125567535: Sending  { Cmd , MgmtId: 7484181839895, via: 3, Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"vmName":"i-2-59-VM","wait":0}}] }
> 2013-10-30 10:22:35,406 DEBUG [agent.transport.Request] (Job-Executor-93:job-244 = [ 539bf83c-67bb-4d4a-a8e2-837a8d85cb4d ]) Seq 3-125567535: Executing:  { Cmd , MgmtId: 7484181839895, via: 3, Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"vmName":"i-2-59-VM","wait":0}}] }
> 2013-10-30 10:22:36,050 DEBUG [xen.resource.CitrixResourceBase] (DirectAgent-68:null) 9. The VM i-2-59-VM is in Stopping state
> 013-10-30 10:22:39,747 DEBUG [cloud.vm.VirtualMachineManagerImpl] (DirectAgent-490:null) VM i-2-59-VM: cs state = Stopping and realState = Running
> 2013-10-30 10:22:39,747 DEBUG [cloud.vm.VirtualMachineManagerImpl] (DirectAgent-490:null) VM i-2-59-VM: cs state = Stopping and realState = Running
> 2013-10-30 10:32:36,831 WARN  [xen.resource.CitrixResourceBase] (DirectAgent-68:null) Async 600 seconds timeout for task com.xensource.xenapi.Task@53bab0c6
> 2013-10-30 10:32:36,843 DEBUG [xen.resource.CitrixResourceBase] (DirectAgent-68:null) Unable to cleanShutdown VM(i-2-59-VM) on host(7f8a4f8f-1c15-4b7a-95cf-1f64b09ce674) due to Async 600 seconds timeout for task com.xensource.xenapi.Task@53bab0c6
> 2013-10-30 10:42:05,852 DEBUG [agent.transport.Request] (HA-Worker-4:work-40) Seq 3-125567622: Sending  { Cmd , MgmtId: 7484181839895, via: 3, Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"vmName":"i-2-59-VM","wait":0}}] }
> 2013-10-30 10:42:05,853 DEBUG [agent.transport.Request] (HA-Worker-4:work-40) Seq 3-125567622: Executing:  { Cmd , MgmtId: 7484181839895, via: 3, Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"vmName":"i-2-59-VM","wait":0}}] }
> 013-10-30 10:42:05,991 DEBUG [xen.resource.CitrixResourceBase] (DirectAgent-218:null) 9. The VM i-2-59-VM is in Stopping state
> 2013-10-30 10:42:05,997 WARN  [xen.resource.CitrixResourceBase] (DirectAgent-152:null) VM destroy failed in Stop i-2-67-VM Command due to You gave an invalid object reference.  The object may have recently been deleted.  The class parameter gives the type of reference given, and the handle parameter echoes the bad value given.
> You gave an invalid object reference.  The object may have recently been deleted.  The class parameter gives the type of reference given, and the handle parameter echoes the bad value given.
>         at com.xensource.xenapi.Types.checkResponse(Types.java:209)
>         at com.xensource.xenapi.Connection.dispatch(Connection.java:368)
>         at com.cloud.hypervisor.xen.resource.XenServerConnectionPool$XenServerConnection.dispatch(XenServerConnectionPool.java:909)
>         at com.xensource.xenapi.VM.getPowerState(VM.java:746)
>         at com.cloud.hypervisor.xen.resource.CitrixResourceBase.execute(CitrixResourceBase.java:4011)
>         at com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:500)
>         at com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:59)
>         at com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:106)
>         at com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>         at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>         at java.lang.Thread.run(Thread.java:722)
> 2013-10-30 10:42:06,122 WARN  [xen.resource.CitrixResourceBase] (DirectAgent-68:null) VM destroy failed in Stop i-2-59-VM Command due to You gave an invalid object reference.  The object may have recently been deleted.  The class parameter gives the type of reference given, and the handle parameter echoes the bad value given.
> You gave an invalid object reference.  The object may have recently been deleted.  The class parameter gives the type of reference given, and the handle parameter echoes the bad value given.
>         at com.xensource.xenapi.Types.checkResponse(Types.java:209)
> 2013-10-30 10:42:06,127 DEBUG [xen.resource.CitrixResourceBase] (DirectAgent-68:null) 10. The VM i-2-59-VM is in Running state
> 2013-10-30 10:42:06,127 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-68:null) Seq 3-125567535: Response Received:


--
This message was sent by Atlassian JIRA
(v6.1.5#6160)