cloudstack-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chandan Purushothama (JIRA)" <j...@apache.org>
Subject [jira] [Created] (CLOUDSTACK-3248) XenServer Host got removed successfully inspite of running VMs on the host
Date Thu, 27 Jun 2013 21:36:20 GMT
Chandan Purushothama created CLOUDSTACK-3248:
------------------------------------------------

             Summary: XenServer Host got removed successfully inspite of running VMs on the
host
                 Key: CLOUDSTACK-3248
                 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-3248
             Project: CloudStack
          Issue Type: Bug
      Security Level: Public (Anyone can view this level - this is the default.)
          Components: Management Server
    Affects Versions: 4.2.0
            Reporter: Chandan Purushothama
            Priority: Critical
             Fix For: 4.2.0


==============
>From the API log:
==============

2013-06-27 11:48:36,855 INFO [cloud.api.ApiServer] (catalina-exec-15:null) (userId=2 accountId=2
sessionId=62A69AC91786781EA9BC2F5E534F5590) 10.252.120.177 -- GET command=deleteHost&id=405585d6-3579-4c8b-82a0-425158ac061a&forced=false&response=json&sessionkey=LAHSBLWAfVc8ti8IX%2FCBeUyvaJQ%3D&_=1372358962625
200 { "deletehostresponse" : { "success" : "true"} }

==========================
>From the Management Server Log:
==========================

2013-06-27 11:48:36,765 DEBUG [cloud.api.ApiServlet] (catalina-exec-15:null) ===START=== 10.252.120.177
-- GET command=deleteHost&id=405585d6-3579-4c8b-82a0-425158ac061a&forced=false&response=json&sessionkey=LAHSBLWAfVc8ti8IX%2FCBeUyvaJQ%3D&_=1372358962625
2013-06-27 11:48:36,800 DEBUG [cloud.resource.ResourceManagerImpl] (catalina-exec-15:null)
Dispatching resource state event DELETE_HOST to ConsoleProxyManagerImpl_EnhancerByCloudStack_edb8e808
2013-06-27 11:48:36,800 DEBUG [cloud.resource.ResourceManagerImpl] (catalina-exec-15:null)
Dispatching resource state event DELETE_HOST to PremiumSecondaryStorageManagerImpl_EnhancerByCloudStack_9f8bf2f0
2013-06-27 11:48:36,800 DEBUG [cloud.resource.ResourceManagerImpl] (catalina-exec-15:null)
Dispatching resource state event DELETE_HOST to JuniperSRXExternalFirewallElement_EnhancerByCloudStack_c0d55fca
2013-06-27 11:48:36,801 DEBUG [cloud.resource.ResourceManagerImpl] (catalina-exec-15:null)
Dispatching resource state event DELETE_HOST to KvmServerDiscoverer_EnhancerByCloudStack_6a93cc3b
2013-06-27 11:48:36,801 DEBUG [cloud.resource.ResourceManagerImpl] (catalina-exec-15:null)
Dispatching resource state event DELETE_HOST to VmwareServerDiscoverer_EnhancerByCloudStack_c0f46d01
2013-06-27 11:48:36,801 DEBUG [cloud.resource.ResourceManagerImpl] (catalina-exec-15:null)
Dispatching resource state event DELETE_HOST to NetscalerElement_EnhancerByCloudStack_2c0cf22
2013-06-27 11:48:36,801 DEBUG [cloud.resource.ResourceManagerImpl] (catalina-exec-15:null)
Dispatching resource state event DELETE_HOST to NetworkUsageManagerImpl_EnhancerByCloudStack_fed7434f
2013-06-27 11:48:36,801 DEBUG [cloud.resource.ResourceManagerImpl] (catalina-exec-15:null)
Dispatching resource state event DELETE_HOST to F5ExternalLoadBalancerElement_EnhancerByCloudStack_91202979
2013-06-27 11:48:36,801 DEBUG [cloud.resource.ResourceManagerImpl] (catalina-exec-15:null)
Dispatching resource state event DELETE_HOST to NiciraNvp
2013-06-27 11:48:36,801 DEBUG [cloud.resource.ResourceManagerImpl] (catalina-exec-15:null)
Dispatching resource state event DELETE_HOST to CiscoVnmcElement_EnhancerByCloudStack_439d7cf0
2013-06-27 11:48:36,801 DEBUG [cloud.resource.ResourceManagerImpl] (catalina-exec-15:null)
Dispatching resource state event DELETE_HOST to OvmDiscoverer_EnhancerByCloudStack_d6ea6366
2013-06-27 11:48:36,802 DEBUG [cloud.resource.ResourceManagerImpl] (catalina-exec-15:null)
Dispatching resource state event DELETE_HOST to LxcServerDiscoverer_EnhancerByCloudStack_2f0a46f8
2013-06-27 11:48:36,801 DEBUG [cloud.resource.ResourceManagerImpl] (catalina-exec-15:null)
Dispatching resource state event DELETE_HOST to OvmDiscoverer_EnhancerByCloudStack_d6ea6366
2013-06-27 11:48:36,802 DEBUG [cloud.resource.ResourceManagerImpl] (catalina-exec-15:null)
Dispatching resource state event DELETE_HOST to LxcServerDiscoverer_EnhancerByCloudStack_2f0a46f8
2013-06-27 11:48:36,802 DEBUG [cloud.resource.ResourceManagerImpl] (catalina-exec-15:null)
Dispatching resource state event DELETE_HOST to XcpServerDiscoverer_EnhancerByCloudStack_6d80ebbf
2013-06-27 11:48:36,802 DEBUG [cloud.resource.ResourceManagerImpl] (catalina-exec-15:null)
Deleting Host: 5 Guid:b213f6c1-16d6-4fa3-8998-dce12a65d1e3
2013-06-27 11:48:36,806 DEBUG [dc.dao.DataCenterIpAddressDaoImpl] (catalina-exec-15:null)
Releasing ip address: 10.223.57.194 data center 1
2013-06-27 11:48:36,810 DEBUG [cloud.host.Status] (catalina-exec-15:null) Transition:[Resource
state = Enabled, Agent event = Remove, Host id = 5, name = Rack3Host8.lab.vmops.com]
2013-06-27 11:48:36,812 DEBUG [cloud.host.Status] (catalina-exec-15:null) Agent status update:
[id = 5; name = Rack3Host8.lab.vmops.com; old status = Disconnected; event = Remove; new status
= Removed; old update count = 12; new update count = 13]
2013-06-27 11:48:36,847 DEBUG [cloud.resource.ResourceManagerImpl] (catalina-exec-15:null)
Cannot transmit host 5to Enabled state
com.cloud.utils.fsm.NoTransitionException: No next resource state found for current state
=Enabled event =DeleteHost
        at com.cloud.resource.ResourceManagerImpl.resourceStateTransitTo(ResourceManagerImpl.java:1367)
        at com.cloud.resource.ResourceManagerImpl.doDeleteHost(ResourceManagerImpl.java:999)
        at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
        at com.cloud.resource.ResourceManagerImpl.deleteHost(ResourceManagerImpl.java:1055)
        at org.apache.cloudstack.api.command.admin.host.DeleteHostCmd.execute(DeleteHostCmd.java:83)
        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:155)
        at com.cloud.api.ApiServer.queueCommand(ApiServer.java:528)
        at com.cloud.api.ApiServer.handleRequest(ApiServer.java:371)
        at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:304)
        at com.cloud.api.ApiServlet.doGet(ApiServlet.java:66)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
        at org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:889)
        at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:721)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2268)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:679)
2013-06-27 11:48:36,855 DEBUG [cloud.api.ApiServlet] (catalina-exec-15:null) ===END=== 10.252.120.177
-- GET command=deleteHost&id=405585d6-3579-4c8b-82a0-425158ac061a&forced=false&response=json&sessionkey=LAHSBLWAfVc8ti8IX%2FCBeUyvaJQ%3D&_=1372358962625

===============
>From the Database:
===============

mysql> select id,name,status,uuid,type,private_ip_address,created,removed from host where
type="Routing";
+----+---------------------------+---------+--------------------------------------+---------+--------------------+---------------------+---------------------+
| id | name | status | uuid | type | private_ip_address | created | removed |
+----+---------------------------+---------+--------------------------------------+---------+--------------------+---------------------+---------------------+
| 1 | Rack3Host20.lab.vmops.com | Up | 9b22f04f-f59d-4145-bddb-138027e4cf56 | Routing | 10.223.59.66
| 2013-06-27 01:08:20 | NULL |
| 5 | Rack3Host8.lab.vmops.com | Removed | 405585d6-3579-4c8b-82a0-425158ac061a | Routing
| 10.223.57.194 | 2013-06-27 01:27:34 | 2013-06-27 18:48:36 |
+----+---------------------------+---------+--------------------------------------+---------+--------------------+---------------------+---------------------+
2 rows in set (0.00 sec) 

mysql> select id,name,instance_name,state from vm_instance where host_id=5;
+----+------------+---------------+---------+
| id | name       | instance_name | state   |
+----+------------+---------------+---------+
|  4 | san        | i-3-4-NTIERR  | Running |
|  5 | r-5-NTIERR | r-5-NTIERR    | Running |
|  6 | san-5      | i-3-6-NTIERR  | Running |
|  7 | san-4      | i-3-7-NTIERR  | Running |
|  8 | san-3      | i-3-8-NTIERR  | Running |
|  9 | san-1      | i-3-9-NTIERR  | Running |
| 10 | san-2      | i-3-10-NTIERR | Running |
| 11 | san-13     | i-3-11-NTIERR | Running |
| 14 | san-12     | i-3-14-NTIERR | Running |
| 15 | san-19     | i-3-15-NTIERR | Running |
| 17 | san-7      | i-3-17-NTIERR | Running |
| 18 | san-16     | i-3-18-NTIERR | Running |
| 21 | san-11     | i-3-21-NTIERR | Running |
+----+------------+---------------+---------+
13 rows in set (0.00 sec)

Any Agent Operation on the above VMs fail

2013-06-27 12:12:12,071 DEBUG [cloud.api.ApiServlet] (catalina-exec-24:null) ===START=== 
10.252.120.177 -- GET  command=destroyVirtualMachine&id=c5e5a1c2-f830-4f0b-8250-e548b48681a1&response=json&sessionkey=LAHSBLWAfVc8ti8IX%2FCBeUyvaJQ%3D&_=1372360378188
2013-06-27 12:12:12,157 DEBUG [cloud.async.AsyncJobManagerImpl] (catalina-exec-24:null) submit
async job-35, details: AsyncJobVO {id:35, userId: 2, accountId: 2, sessionKey: null, instanceType:
VirtualMachine, instanceId: 4, cmd: org.apache.cloudstack.api.command.user.vm.DestroyVMCmd,
cmdOriginator: null, cmdInfo: {"id":"c5e5a1c2-f830-4f0b-8250-e548b48681a1","response":"json","sessionkey":"LAHSBLWAfVc8ti8IX/CBeUyvaJQ\u003d","ctxUserId":"2","httpmethod":"GET","_":"1372360378188","ctxAccountId":"2","ctxStartEventId":"170"},
cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0, processStatus: 0, resultCode:
0, result: null, initMsid: 7471666038533, completeMsid: null, lastUpdated: null, lastPolled:
null, created: null}
2013-06-27 12:12:12,159 DEBUG [cloud.api.ApiServlet] (catalina-exec-24:null) ===END===  10.252.120.177
-- GET  command=destroyVirtualMachine&id=c5e5a1c2-f830-4f0b-8250-e548b48681a1&response=json&sessionkey=LAHSBLWAfVc8ti8IX%2FCBeUyvaJQ%3D&_=1372360378188
2013-06-27 12:12:12,182 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-1:job-35) Executing
org.apache.cloudstack.api.command.user.vm.DestroyVMCmd for job-35
2013-06-27 12:12:12,253 DEBUG [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-1:job-35)
Destroying vm VM[User|san]
2013-06-27 12:12:12,285 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-1:job-35)
VM state transitted from :Running to Stopping with event: StopRequestedvm's original host
id: 5 new host id: 5 host id before state transition: 5
2013-06-27 12:12:12,288 WARN  [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-1:job-35)
Unable to stop vm, agent unavailable: com.cloud.exception.AgentUnavailableException: Resource
[Host:5] is unreachable: Host 5: Can't find the host
2013-06-27 12:12:12,288 WARN  [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-1:job-35)
Unable to stop vm VM[User|san]
2013-06-27 12:12:12,294 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-1:job-35)
VM state transitted from :Stopping to Running with event: OperationFailedvm's original host
id: 5 new host id: 5 host id before state transition: 5
2013-06-27 12:12:12,294 DEBUG [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-1:job-35)
Unable to stop VM[User|san]
2013-06-27 12:12:12,300 ERROR [cloud.async.AsyncJobManagerImpl] (Job-Executor-1:job-35) Unexpected
exception while executing org.apache.cloudstack.api.command.user.vm.DestroyVMCmd
com.cloud.utils.exception.CloudRuntimeException: Failed to destroy vm with specified vmId
        at com.cloud.vm.UserVmManagerImpl.destroyVm(UserVmManagerImpl.java:3386)
        at com.cloud.vm.UserVmManagerImpl.destroyVm(UserVmManagerImpl.java:1881)
        at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
        at org.apache.cloudstack.api.command.user.vm.DestroyVMCmd.execute(DestroyVMCmd.java:100)
        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:155)
        at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:437)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:679)
2013-06-27 12:12:12,300 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-1:job-35) Complete
async job-35, jobStatus: 2, resultCode: 530, result: Error Code: 530 Error text: Failed to
destroy vm with specified vmId


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message