cloudstack-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Abhinav Roy (JIRA)" <j...@apache.org>
Subject [jira] [Created] (CLOUDSTACK-4506) In a mixed hypervisor setup, destroying a VM whose host has been removed, throws a NPE and the ROOT volume of that VM also is not deleted from the primary.
Date Mon, 26 Aug 2013 14:53:53 GMT
Abhinav Roy created CLOUDSTACK-4506:
---------------------------------------

             Summary: In a mixed hypervisor setup, destroying a VM whose host has been removed,
throws a NPE and the ROOT volume of that VM also is not deleted from the primary.
                 Key: CLOUDSTACK-4506
                 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4506
             Project: CloudStack
          Issue Type: Bug
      Security Level: Public (Anyone can view this level - this is the default.)
          Components: Storage Controller
    Affects Versions: 4.2.0
         Environment: Advanced zone setup having clusters of different hypervisor types. Ex
KVM and VMWARE
            Reporter: Abhinav Roy
            Priority: Critical
             Fix For: 4.2.1


Steps :
=========================
1. Deploy a CS 4.2 setup with KVM and VMWARE clusters having one host each.
2. Create some VMs on KVM (ex- kvm1 and kvm2)
3. Put the KVM host in maintenance mode and then remove the Host. Now kvm1 and kvm2 are in
stopped state.
4. Destroy kvm1 


Observations :
========================
1. when kvm1 is destroyed it fails with the following exception 

2013-08-26 13:10:09,205 DEBUG [cloud.api.ApiServlet] (catalina-exec-19:null) ===START=== 
10.144.6.17 -- GET  command=destroyVirtualMachine&id=d613f6e5-c53a-4f6b-be31-32f101eb6c99&response=json&sessionkey=bU49tWdfVUJUrq65PbMmzGbe0PE%3D&_=1377502661180
2013-08-26 13:10:09,242 DEBUG [cloud.async.AsyncJobManagerImpl] (catalina-exec-19:null) submit
async job-78 = [ 8018e79c-a47e-4a54-b3a4-68963f3c11a9 ], details: AsyncJobVO {id:78, userId:
2, accountId: 2, sessionKey: null, instanceType: VirtualMachine, instanceId: 3, cmd: org.apache.cloudstack.api.command.user.vm.DestroyVMCmd,
cmdOriginator: null, cmdInfo: {"response":"json","id":"d613f6e5-c53a-4f6b-be31-32f101eb6c99","sessionkey":"bU49tWdfVUJUrq65PbMmzGbe0PE\u003d","cmdEventType":"VM.DESTROY","ctxUserId":"2","httpmethod":"GET","_":"1377502661180","ctxAccountId":"2","ctxStartEventId":"245"},
cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0, processStatus: 0, resultCode:
0, result: null, initMsid: 226870599129537, completeMsid: null, lastUpdated: null, lastPolled:
null, created: null}
2013-08-26 13:10:09,245 DEBUG [cloud.api.ApiServlet] (catalina-exec-19:null) ===END===  10.144.6.17
-- GET  command=destroyVirtualMachine&id=d613f6e5-c53a-4f6b-be31-32f101eb6c99&response=json&sessionkey=bU49tWdfVUJUrq65PbMmzGbe0PE%3D&_=1377502661180
2013-08-26 13:10:09,249 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-50:job-78 =
[ 8018e79c-a47e-4a54-b3a4-68963f3c11a9 ]) Executing org.apache.cloudstack.api.command.user.vm.DestroyVMCmd
for job-78 = [ 8018e79c-a47e-4a54-b3a4-68963f3c11a9 ]
2013-08-26 13:10:09,282 DEBUG [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-50:job-78
= [ 8018e79c-a47e-4a54-b3a4-68963f3c11a9 ]) Destroying vm VM[User|v1]
2013-08-26 13:10:09,282 DEBUG [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-50:job-78
= [ 8018e79c-a47e-4a54-b3a4-68963f3c11a9 ]) VM is already stopped: VM[User|v1]
2013-08-26 13:10:09,308 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-50:job-78
= [ 8018e79c-a47e-4a54-b3a4-68963f3c11a9 ]) VM state transitted from :Stopped to Destroyed
with event: DestroyRequestedvm's original host id: 1 new host id: null host id before state
transition: null
2013-08-26 13:10:09,324 ERROR [cloud.async.AsyncJobManagerImpl] (Job-Executor-50:job-78 =
[ 8018e79c-a47e-4a54-b3a4-68963f3c11a9 ]) Unexpected exception while executing org.apache.cloudstack.api.command.user.vm.DestroyVMCmd
java.lang.NullPointerException
        at com.cloud.capacity.CapacityManagerImpl.releaseVmCapacity(CapacityManagerImpl.java:187)
        at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
        at com.cloud.capacity.CapacityManagerImpl.postStateTransitionEvent(CapacityManagerImpl.java:718)
        at com.cloud.capacity.CapacityManagerImpl.postStateTransitionEvent(CapacityManagerImpl.java:101)
        at com.cloud.utils.fsm.StateMachine2.transitTo(StateMachine2.java:117)
        at com.cloud.vm.VirtualMachineManagerImpl.stateTransitTo(VirtualMachineManagerImpl.java:1324)
        at com.cloud.vm.VirtualMachineManagerImpl.destroy(VirtualMachineManagerImpl.java:1355)
        at org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.destroyVirtualMachine(VMEntityManagerImpl.java:259)
        at org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.destroy(VirtualMachineEntityImpl.java:225)
        at com.cloud.vm.UserVmManagerImpl.destroyVm(UserVmManagerImpl.java:3450)
        at com.cloud.vm.UserVmManagerImpl.destroyVm(UserVmManagerImpl.java:1986)
        at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
        at org.apache.cloudstack.api.command.user.vm.DestroyVMCmd.execute(DestroyVMCmd.java:100)
        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
        at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:636)
2013-08-26 13:10:09,328 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-50:job-78 =
[ 8018e79c-a47e-4a54-b3a4-68963f3c11a9 ]) Complete async job-78 = [ 8018e79c-a47e-4a54-b3a4-68963f3c11a9
], jobStatus: 2, resultCode: 530, result: Error Code: 530 Error text: null


2. The volume of that vm is not removed from the primary

mysql> SELECT * FROM cloud.volumes where id=16;
+----+------------+-----------+---------+--------------+-------------+-----------+--------+--------------------------------------+------------+---------------------------+--------------------------------------+--------+----------------+------------+---------+-------------+-----------+------------------+-------------+----------------------------+-------------+---------------------+----------+---------------------+---------+-----------+------------+--------------+-----------+------------------------+--------+----------------+--------+----------+----------+
| id | account_id | domain_id | pool_id | last_pool_id | instance_id | device_id | name  
| uuid                                 | size       | folder                    | path   
                             | pod_id | data_center_id | iscsi_name | host_ip | volume_type
| pool_type | disk_offering_id | template_id | first_snapshot_backup_uuid | recreatable |
created             | attached | updated             | removed | state     | chain_info |
update_count | disk_type | vm_snapshot_chain_size | iso_id | display_volume | format | min_iops
| max_iops |
+----+------------+-----------+---------+--------------+-------------+-----------+--------+--------------------------------------+------------+---------------------------+--------------------------------------+--------+----------------+------------+---------+-------------+-----------+------------------+-------------+----------------------------+-------------+---------------------+----------+---------------------+---------+-----------+------------+--------------+-----------+------------------------+--------+----------------+--------+----------+----------+
| 16 |          2 |         1 |       7 |            1 |           6 |         0 | ROOT-6
| 9ba1da18-bc84-4c6c-9fdd-107b89e09a03 | 8589934592 | /cpg_vol/abhinav/kvm-pri2 | ab000d45-4185-4347-9c7a-68e8a5ebbb49
|      1 |              1 | NULL       | NULL    | ROOT        | NULL      |             
  1 |           4 | NULL                       |           0 | 2013-08-23 12:01:55 | NULL
    | 2013-08-26 07:40:46 | NULL    | Expunging | NULL       |            6 | NULL      |
                  NULL |      0 |              0 | NULL   |     NULL |     NULL |
+----+------------+-----------+---------+--------------+-------------+-----------+--------+--------------------------------------+------------+---------------------------+--------------------------------------+--------+----------------+------------+---------+-------------+-----------+------------------+-------------+----------------------------+-------------+---------------------+----------+---------------------+---------+-----------+------------+--------------+-----------+------------------------+--------+----------------+--------+----------+----------+
1 row in set (0.00 sec)


[root@MS-CentOS60 mnt]# ls primary/
ab000d45-4185-4347-9c7a-68e8a5ebbb49  KVMFILELOCK  KVMHA
[root@MS-CentOS60 mnt]# ll primary/
total 1605080
-rwxr--r--. 1 root root 1640366080 Aug 26 12:29 ab000d45-4185-4347-9c7a-68e8a5ebbb49
drwxr-xr-x. 2 root root       4096 Aug 26 12:28 KVMFILELOCK
drwxr-xr-x. 2 root root       4096 Aug 23 17:29 KVMHA


3. This is applicable for all vms, user vms and system vms

Attaching management server logs and db dump.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message