cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Denis Finko <denis.fi...@ecommerce.com>
Subject Re: Primary Storage Allocated issue
Date Fri, 17 Oct 2014 11:48:39 GMT
GopalaKrishnan, I am appreciate you for quick reply!

But unfortunately "expunge.interval" is not helpful. I have several time updated it recently
and it wasn't removed old VMs from VMware.

Currently I have:
expunge.interval = 86400

> There is no need to push manually to remove expunged volumes.
Yes, I am agree with you that it's supposed to be. But unfortunately I still have a lot of
volumes in 'Expunged' state and don't know how correctly remove them from CloudStack side.
It looks like a bug in CloudStack.


On 10/17/2014 01:35 PM, GopalaKrishnan wrote:
> Hi Denis,
>
> I think, it should be calculated disk size also from 'Expunged' volumes. Try to update
"expunge.interval" in cloudstack global settings.
>
> There is no need to push manually to remove expunged volumes.
>
> Thank You.
> Gopalakrishnan.S
> Cloud Consultant.
> ----- Original Message ----- From: "Denis Finko" <denis.finko@ecommerce.com>
> To: <dev@cloudstack.apache.org>; <users@cloudstack.apache.org>
> Sent: Friday, October 17, 2014 1:47 PM
> Subject: Primary Storage Allocated issue
>
>
>> Hello CloudStack community,
>>
>> Could you please take a look to following issue. The similar was described it that
topic http://comments.gmane.org/gmane.comp.apache.cloudstack.user/13848 but looks like wasn't
resolved.
>>
>> In my environment I have:
>> CloudStack v. 4.2.1
>> VMware v. 5.1
>>
>> Recently I have found that CloudStack UI provide following information for SATA storage:
>> Disk Total           5.00 TB
>> Disk Allocated    7.26 TB
>>
>> I have looked to VMware side and found absolutely different data:
>> Capacity:        5.00 TB
>> Provisioned Space:    4.27 TB
>>
>> As you can see CloudStack show allocated disk space on 3 TB more.
>>
>> My Global Settings that could be helpful for investigation:
>> storage.overprovisioning.factor = 4
>> storage.cleanup.enabled = true
>> storage.cleanup.interval = 86400
>> storage.stats.interval = 60000
>> capacity.check.period = 60000
>> expunge.delay    =    60
>> expunge.interval    =    86400
>> expunge.workers    =    10
>>
>> I have looked to CloudStack database and found that some Volumes state in 'Destroy'
but also I have found a lot of in 'Expunged':
>> mysql> select v.state, sum(v.size) from volumes v, vm_instance vm, vm_root_disk_tags
d where v.instance_id=vm.id and d.vm_id=vm.id and d.root_disk_tag like 'sata' group by v.state;
>> +-----------+---------------+
>> | state             | sum(v.size)   |
>> +-----------+---------------+
>> | Destroy      |  664646189056 |
>> | Expunged  | 5267777388544 |
>> | Expunging  |    2147483648 |
>> | Ready         | 3833258311680 |
>> +-----------+---------------+
>> 4 rows in set (0.00 sec)
>>
>> And a lot of VMs that have state=Expunged (in volume table) still present in VMware
and on storage. For example all these VMs still present in VMware:
>>
>> mysql> select vm.name,vm.instance_name,vm.state AS VM_state,v.id,v.size,v.state
AS VOLUME_state,v.removed,d.root_disk_tag from volumes v, vm_instance vm, vm_root_disk_tags
d where v.instance_id=vm.id and d.vm_id=vm.id and v.state like 'Expunged' and d.root_disk_tag
like 'sata'; (Names have been
>> changed)
>> +----------+---------------+-----------+-----------+--------------+--------------+---------------------+---------------+
>> | name     | instance_name |    VM_state      |    id           |    size | VOLUME_state
| removed             | root_disk_tag |
>> +----------+---------------+-----------+-----------+--------------+--------------+---------------------+---------------+
>> | name1    | i-43-2147-VM  | Expunging |      2195 |  53687091200 | Expunged    
| 2014-10-16 08:12:04 | sata          |
>> | name2    | i-17-2123-VM  | Expunging |      2173 |  10737418240 | Expunged    
| 2014-10-15 19:14:30 | sata          |
>> | name3    | i-43-2052-VM  | Expunging |      2098 |  10737418240 | Expunged    
| 2014-10-07 11:57:49 | sata          |
>> | name4    | i-191-1954-VM | Expunging |      2000 |  42949672960 | Expunged    
| 2014-10-01 15:15:04 | sata          |
>> | name5    | i-132-1966-VM | Expunging |      2012 |  21474836480 | Expunged    
| 2014-09-29 20:33:12 | sata          |
>> | name6    | i-132-1967-VM | Expunging |      2013 |  21474836480 | Expunged    
| 2014-09-29 20:33:12 | sata          |
>> | name7    | i-132-1962-VM | Expunging |      2008 |  10737418240 | Expunged    
| 2014-09-29 20:33:11 | sata          |
>> | name8    | i-132-1964-VM | Expunging |      2010 |  10737418240 | Expunged    
| 2014-09-29 20:33:11 | sata          |
>>
>> Not all VMs with 'Expunged' state still present in VMware but a lot of from them
still weren't removed!
>>
>> mysql> select id,uuid,name,status,capacity_bytes,capacity_iops,tag,disk_used_capacity,disk_reserved_capacity
from storage_pool_view;
>> +----+--------------------------------------------------------------+---------------+-------------+----------------+---------------+------+--------------------+------------------------+
>> | id | uuid | name          | status      | capacity_bytes | capacity_iops | tag
 | disk_used_capacity | disk_reserved_capacity |
>> +----+--------------------------------------------------------------+---------------+-------------+----------------+---------------+------+--------------------+------------------------+
>> |  2 | 096b3b6e-481d-33b4-9b34-7178612a2535  |     SATA      | Up |  5497289703424
|          NULL | sata |      7913594810761 | 0 |
>> +----+--------------------------------------------------------------+---------------+-------------+----------------+---------------+------+--------------------+------------------------+
>> 5 rows in set (0.02 sec)
>>
>> Also I have found that Expunged volumes have 'chain_info' field:
>>
>> mysql> select name,removed,state,chain_info from volumes where id=2112 \G
>> *************************** 1. row ***************************
>>      name: ROOT-2066
>>   removed: 2014-10-09 11:57:52
>>     state: Expunged
>> chain_info: {"diskDeviceBusName":"scsi0:0","diskChain":["[SATA1] i-238-2066-VM/ROOT-2066.vmdk"]}
>> 1 row in set (0.00 sec)
>>
>>
>> BUT Destroyed VMs don't have it:
>> mysql> select name,removed,state,chain_info from volumes where id=2192 \G
>> *************************** 1. row ***************************
>>      name: ROOT-2143
>>   removed: 2014-10-15 18:42:48
>>     state: Destroy
>> chain_info: NULL
>> 1 row in set (0.00 sec)
>>
>>
>> So, the questions are:
>> - How CloudStack calculate allocated disk size for storage? Is it use disk size from
'Expunged' volumes?
>> - Can I manually push CloudStack to remove 'Expunged' volumes from VMware?
>>
>> I haven't created bug in Jira but I can if it's needed.
>>
>> Thank you!
>>
>> -- 
>> Best regards,
>> Denis
>>
>
>
> ---
> This email is free from viruses and malware because avast! Antivirus protection is active.
> http://www.avast.com
>
>

-- 
Best regards,
Denis


Mime
View raw message