Mailing-List: contact dev-help@cloudstack.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cloudstack.apache.org
Received-SPF: unknown (athena.apache.org: error in processing during lookup of
 fgaudreault@cloudops.com)
Message-ID: <53FC99BA.3010909@cloudops.com>
Date: Tue, 26 Aug 2014 10:29:14 -0400
From: Francois Gaudreault <fgaudreault@cloudops.com>
Reply-To: fgaudreault@cloudops.com
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9;
 rv:24.0) Gecko/20100101 Thunderbird/24.6.0
MIME-Version: 1.0
To: "dev@cloudstack.apache.org" <dev@cloudstack.apache.org>
Subject: VMs stuck in expunging
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Hi,

We have couple VMs stuck in Expunging state in our setup, and I am not 
sure why. This is a 4.3.1-snapshot. This is what the log says:

2014-08-25 23:30:31,789 DEBUG [c.c.v.VirtualMachineManagerImpl] 
(UserVm-Scavenger-1:ctx-bfdf6aa3) Stopped called on VM[User|ketchup-web] 
but the state is Expunging
2014-08-25 23:30:31,808 WARN  [c.c.u.n.Link] 
(AgentManager-Selector:null) SSL: Fail to find the generated keystore. 
Loading fail-safe one to continue.
2014-08-25 23:30:31,810 DEBUG [c.c.c.CapacityManagerImpl] 
(UserVm-Scavenger-1:ctx-bfdf6aa3) VM state transitted from :Expunging to 
Expunging with event: ExpungeOperationvm's original host id: 1 new host 
id: null host id before state transition: null
2014-08-25 23:30:31,811 DEBUG [c.c.v.VirtualMachineManagerImpl] 
(UserVm-Scavenger-1:ctx-bfdf6aa3) Destroying vm VM[User|ketchup-web]
2014-08-25 23:30:31,811 DEBUG [c.c.v.VirtualMachineManagerImpl] 
(UserVm-Scavenger-1:ctx-bfdf6aa3) Cleaning up NICS
2014-08-25 23:30:31,811 DEBUG [o.a.c.e.o.NetworkOrchestrator] 
(UserVm-Scavenger-1:ctx-bfdf6aa3) Cleaning network for vm: 61
2014-08-25 23:30:31,814 DEBUG [c.c.v.VirtualMachineManagerImpl] 
(UserVm-Scavenger-1:ctx-bfdf6aa3) Cleaning up hypervisor data structures 
(ex. SRs in XenServer) for managed storage
2014-08-25 23:30:31,821 DEBUG [o.a.c.e.o.VolumeOrchestrator] 
(UserVm-Scavenger-1:ctx-bfdf6aa3) Cleaning storage for vm: 61
2014-08-25 23:30:31,825 DEBUG [c.c.v.VirtualMachineManagerImpl] 
(UserVm-Scavenger-1:ctx-bfdf6aa3) Expunged VM[User|ketchup-web]
2014-08-25 23:30:31,825 DEBUG [c.c.v.UserVmManagerImpl] 
(UserVm-Scavenger-1:ctx-bfdf6aa3) Starting cleaning up vm 
VM[User|ketchup-web] resources...
2014-08-25 23:30:31,856 DEBUG [c.c.n.f.FirewallManagerImpl] 
(UserVm-Scavenger-1:ctx-bfdf6aa3) No firewall rules are found for vm id=61
2014-08-25 23:30:31,872 DEBUG [c.c.v.UserVmManagerImpl] 
(UserVm-Scavenger-1:ctx-bfdf6aa3) Firewall rules are removed 
successfully as a part of vm id=61 expunge
2014-08-25 23:30:31,884 DEBUG [c.c.u.d.T.Transaction] 
(UserVm-Scavenger-1:ctx-bfdf6aa3) Rolling back the transaction: Time = 5 
Name =  UserVm-Scavenger-1; called by 
-TransactionLegacy.rollback:896-TransactionLegacy.removeUpTo:839-TransactionLegacy.close:663-Transaction.execute:41-Transaction.execute:46-FirewallManagerImpl.revokeRule:732-GeneratedMethodAccessor278.invoke:-1-DelegatingMethodAccessorImpl.invoke:43-Method.invoke:606-AopUtils.invokeJoinpointUsingReflection:317-ReflectiveMethodInvocation.invokeJoinpoint:183-ReflectiveMethodInvocation.proceed:150
2014-08-25 23:30:31,887 WARN  [c.c.v.UserVmManagerImpl] 
(UserVm-Scavenger-1:ctx-bfdf6aa3) Unable to expunge VM[User|ketchup-web]

Any idea what's going on? Bug?

-- 
Francois Gaudreault
Gestionnaire de Produit | Product Manager - Cloud Platform & Services
t:514-629-6775

CloudOps Votre partenaire infonuagique | Cloud Solutions Experts
420 rue Guy | Montreal | Quebec | H3J 1S6
w: cloudops.com | tw: @CloudOps_