cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Denis Finko <denis.fi...@ecommerce.com>
Subject Primary Storage Allocated issue
Date Fri, 17 Oct 2014 08:17:42 GMT
Hello CloudStack community,

Could you please take a look to following issue. The similar was described it that topic http://comments.gmane.org/gmane.comp.apache.cloudstack.user/13848
but looks like wasn't resolved.

In my environment I have:
CloudStack v. 4.2.1
VMware v. 5.1

Recently I have found that CloudStack UI provide following information for SATA storage:
Disk Total           5.00 TB
Disk Allocated    7.26 TB

I have looked to VMware side and found absolutely different data:
Capacity:        5.00 TB
Provisioned Space:    4.27 TB

As you can see CloudStack show allocated disk space on 3 TB more.

My Global Settings that could be helpful for investigation:
storage.overprovisioning.factor = 4
storage.cleanup.enabled = true
storage.cleanup.interval = 86400
storage.stats.interval = 60000
capacity.check.period = 60000
expunge.delay    =    60
expunge.interval    =    86400
expunge.workers    =    10

I have looked to CloudStack database and found that some Volumes state in 'Destroy' but also
I have found a lot of in 'Expunged':
mysql> select v.state, sum(v.size) from volumes v, vm_instance vm, vm_root_disk_tags d
where v.instance_id=vm.id and d.vm_id=vm.id and d.root_disk_tag like 'sata' group by v.state;
+-----------+---------------+
| state             | sum(v.size)   |
+-----------+---------------+
| Destroy      |  664646189056 |
| Expunged  | 5267777388544 |
| Expunging  |    2147483648 |
| Ready         | 3833258311680 |
+-----------+---------------+
4 rows in set (0.00 sec)

And a lot of VMs that have state=Expunged (in volume table) still present in VMware and on
storage. For example all these VMs still present in VMware:

mysql> select vm.name,vm.instance_name,vm.state AS VM_state,v.id,v.size,v.state AS VOLUME_state,v.removed,d.root_disk_tag
from volumes v, vm_instance vm, vm_root_disk_tags d where v.instance_id=vm.id and d.vm_id=vm.id
and v.state like 'Expunged' and d.root_disk_tag like 'sata'; (Names have been changed)
+----------+---------------+-----------+-----------+--------------+--------------+---------------------+---------------+
| name     | instance_name |    VM_state      |    id           |    size           | VOLUME_state
| removed             | root_disk_tag |
+----------+---------------+-----------+-----------+--------------+--------------+---------------------+---------------+
| name1    | i-43-2147-VM  | Expunging |      2195 |  53687091200 | Expunged     | 2014-10-16
08:12:04 | sata          |
| name2    | i-17-2123-VM  | Expunging |      2173 |  10737418240 | Expunged     | 2014-10-15
19:14:30 | sata          |
| name3    | i-43-2052-VM  | Expunging |      2098 |  10737418240 | Expunged     | 2014-10-07
11:57:49 | sata          |
| name4    | i-191-1954-VM | Expunging |      2000 |  42949672960 | Expunged     | 2014-10-01
15:15:04 | sata          |
| name5    | i-132-1966-VM | Expunging |      2012 |  21474836480 | Expunged     | 2014-09-29
20:33:12 | sata          |
| name6    | i-132-1967-VM | Expunging |      2013 |  21474836480 | Expunged     | 2014-09-29
20:33:12 | sata          |   
| name7    | i-132-1962-VM | Expunging |      2008 |  10737418240 | Expunged     | 2014-09-29
20:33:11 | sata          |
| name8    | i-132-1964-VM | Expunging |      2010 |  10737418240 | Expunged     | 2014-09-29
20:33:11 | sata          | 

Not all VMs with 'Expunged' state still present in VMware but a lot of from them still weren't
removed!

mysql> select id,uuid,name,status,capacity_bytes,capacity_iops,tag,disk_used_capacity,disk_reserved_capacity
from storage_pool_view;
+----+--------------------------------------------------------------+---------------+-------------+----------------+---------------+------+--------------------+------------------------+
| id | uuid                                                                        | name
         | status      | capacity_bytes | capacity_iops | tag  | disk_used_capacity | disk_reserved_capacity
|
+----+--------------------------------------------------------------+---------------+-------------+----------------+---------------+------+--------------------+------------------------+
|  2 | 096b3b6e-481d-33b4-9b34-7178612a2535  |     SATA      | Up          |  5497289703424
|          NULL | sata |      7913594810761 |                      0 |
+----+--------------------------------------------------------------+---------------+-------------+----------------+---------------+------+--------------------+------------------------+
5 rows in set (0.02 sec)

Also I have found that Expunged volumes have 'chain_info' field:

mysql> select name,removed,state,chain_info from volumes where id=2112 \G
*************************** 1. row ***************************
      name: ROOT-2066
   removed: 2014-10-09 11:57:52
     state: Expunged
chain_info: {"diskDeviceBusName":"scsi0:0","diskChain":["[SATA1] i-238-2066-VM/ROOT-2066.vmdk"]}
1 row in set (0.00 sec)


BUT Destroyed VMs don't have it:
mysql> select name,removed,state,chain_info from volumes where id=2192 \G
*************************** 1. row ***************************
      name: ROOT-2143
   removed: 2014-10-15 18:42:48
     state: Destroy
chain_info: NULL
1 row in set (0.00 sec)


So, the questions are:
- How CloudStack calculate allocated disk size for storage? Is it use disk size from 'Expunged'
volumes?
- Can I manually push CloudStack to remove 'Expunged' volumes from VMware?

I haven't created bug in Jira but I can if it's needed.

Thank you!

-- 
Best regards,
Denis


Mime
View raw message