cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Denis Finko <denis.fi...@ecommerce.com>
Subject Re: Primary Storage Allocated issue
Date Mon, 20 Oct 2014 09:55:09 GMT
Hello

I have tried this and set expunge.interval and expunge.delay to 120 on weekend. But it wasn't
helpful. Old VMs (not all but some of them) still present in VMware environment.
Looks like I am need to create a bug.

Is the any way to correctly remove them from CloudStack?

Just several examples:

mysql> select vm.name,vm.state AS VM_state,v.id as VOLUME_id,v.size,v.state AS VOLUME_state,v.removed,d.root_disk_tag,v.chain_info
from volumes v, vm_instance vm, vm_root_disk_tags d where v.instance_id=vm.id and d.vm_id=vm.id
and vm.instance_name like 'i-132-1967-VM' \G
*************************** 1. row ***************************
         name: Test4UbuntuDesktop14-04
     VM_state: Expunging
    VOLUME_id: 2013
         size: 21474836480
 VOLUME_state: Expunged
      removed: 2014-09-29 20:33:12
root_disk_tag: auto
   chain_info: {"diskDeviceBusName":"scsi0:0","diskChain":["[AUTO1] i-132-1967-VM/4e34b5f5d1214296a762c633507acfd8.vmdk"]}
1 row in set (0.03 sec)

mysql>


mysql> select vm.name,vm.state AS VM_state,v.id as VOLUME_id,v.size,v.state AS VOLUME_state,v.removed,d.root_disk_tag,v.chain_info
from volumes v, vm_instance vm, vm_root_disk_tags d where v.instance_id=vm.id and d.vm_id=vm.id
and vm.instance_name like 'i-161-1922-VM' \G
*************************** 1. row ***************************
         name: FCtest
     VM_state: Expunging
    VOLUME_id: 1970
         size: 42949672960
 VOLUME_state: Expunged
      removed: 2014-09-22 20:33:01
root_disk_tag: auto
   chain_info: {"diskDeviceBusName":"scsi0:0","diskChain":["[AUTO2] i-161-1922-VM/ROOT-1922.vmdk"]}
1 row in set (0.02 sec)

mysql>


mysql> select vm.name,vm.state AS VM_state,v.id as VOLUME_id,v.size,v.state AS VOLUME_state,v.removed,d.root_disk_tag,v.chain_info
from volumes v, vm_instance vm, vm_root_disk_tags d where v.instance_id=vm.id and d.vm_id=vm.id
and vm.instance_name like 'i-2-1630-VM' \G
*************************** 1. row ***************************
         name: myserver
     VM_state: Expunging
    VOLUME_id: 1676
         size: 10737418240
 VOLUME_state: Expunged
      removed: 2014-08-14 11:37:24
root_disk_tag: sata
   chain_info: {"diskDeviceBusName":"scsi0:0","diskChain":["[SATA1] i-2-1630-VM/ROOT-1630.vmdk"]}
1 row in set (0.03 sec)

mysql>


On 10/17/2014 03:11 PM, GopalaKrishnan wrote:
> Hi Denis,
>
> Try to reduce expunge interval values to 120 and expunge delay. When you delete VMs or
disk, etc., CloudStack will keep its details in the database and disks on the primary storage
for 24 hours.
>
> You can shorten this period to expunge 120 seconds.
>
> expunge.interval = 120
> expunge.delay=120
>
> Restart cloudstack once these changes completed.
>
> Thank You.
> Gopalakrishnan.S
> Cloud Consultant - Fogpanel.
>
>
>
> ----- Original Message ----- From: "Denis Finko" <denis.finko@ecommerce.com>
> To: <users@cloudstack.apache.org>; <dev@cloudstack.apache.org>
> Sent: Friday, October 17, 2014 5:21 PM
> Subject: Re: Primary Storage Allocated issue
>
>
>> GopalaKrishnan, I am appreciate you for quick reply!
>>
>> But unfortunately "expunge.interval" is not helpful. I have several time updated
it recently and it wasn't removed old VMs from VMware.
>>
>> Currently I have:
>> expunge.interval = 86400
>>
>>> There is no need to push manually to remove expunged volumes.
>> Yes, I am agree with you that it's supposed to be. But unfortunately I still have
a lot of volumes in 'Expunged' state and don't know how correctly remove them from CloudStack
side. It looks like a bug in CloudStack.
>>
>> On 10/17/2014 01:35 PM, GopalaKrishnan wrote:
>>> Hi Denis,
>>>
>>> I think, it should be calculated disk size also from 'Expunged' volumes. Try
to update "expunge.interval" in cloudstack global settings.
>>>
>>> There is no need to push manually to remove expunged volumes.
>>>
>>> Thank You.
>>> Gopalakrishnan.S
>>> Cloud Consultant.
>>> ----- Original Message ----- From: "Denis Finko" <denis.finko@ecommerce.com>
>>> To: <dev@cloudstack.apache.org>; <users@cloudstack.apache.org>
>>> Sent: Friday, October 17, 2014 1:47 PM
>>> Subject: Primary Storage Allocated issue
>>>
>>>
>>>> Hello CloudStack community,
>>>>
>>>> Could you please take a look to following issue. The similar was described
it that topic http://comments.gmane.org/gmane.comp.apache.cloudstack.user/13848 but looks
like wasn't resolved.
>>>>
>>>> In my environment I have:
>>>> CloudStack v. 4.2.1
>>>> VMware v. 5.1
>>>>
>>>> Recently I have found that CloudStack UI provide following information for
SATA storage:
>>>> Disk Total           5.00 TB
>>>> Disk Allocated    7.26 TB
>>>>
>>>> I have looked to VMware side and found absolutely different data:
>>>> Capacity:        5.00 TB
>>>> Provisioned Space:    4.27 TB
>>>>
>>>> As you can see CloudStack show allocated disk space on 3 TB more.
>>>>
>>>> My Global Settings that could be helpful for investigation:
>>>> storage.overprovisioning.factor = 4
>>>> storage.cleanup.enabled = true
>>>> storage.cleanup.interval = 86400
>>>> storage.stats.interval = 60000
>>>> capacity.check.period = 60000
>>>> expunge.delay    =    60
>>>> expunge.interval    =    86400
>>>> expunge.workers    =    10
>>>>
>>>> I have looked to CloudStack database and found that some Volumes state in
'Destroy' but also I have found a lot of in 'Expunged':
>>>> mysql> select v.state, sum(v.size) from volumes v, vm_instance vm, vm_root_disk_tags
d where v.instance_id=vm.id and d.vm_id=vm.id and d.root_disk_tag like 'sata' group by v.state;
>>>> +-----------+---------------+
>>>> | state             | sum(v.size)   |
>>>> +-----------+---------------+
>>>> | Destroy      |  664646189056 |
>>>> | Expunged  | 5267777388544 |
>>>> | Expunging  |    2147483648 |
>>>> | Ready         | 3833258311680 |
>>>> +-----------+---------------+
>>>> 4 rows in set (0.00 sec)
>>>>
>>>> And a lot of VMs that have state=Expunged (in volume table) still present
in VMware and on storage. For example all these VMs still present in VMware:
>>>>
>>>> mysql> select vm.name,vm.instance_name,vm.state AS VM_state,v.id,v.size,v.state
AS VOLUME_state,v.removed,d.root_disk_tag from volumes v, vm_instance vm, vm_root_disk_tags
d where v.instance_id=vm.id and d.vm_id=vm.id and v.state like 'Expunged' and d.root_disk_tag
like 'sata'; (Names have been
>>>> changed)
>>>> +----------+---------------+-----------+-----------+--------------+--------------+---------------------+---------------+
>>>> | name     | instance_name |    VM_state      |    id           | size |
VOLUME_state | removed             | root_disk_tag |
>>>> +----------+---------------+-----------+-----------+--------------+--------------+---------------------+---------------+
>>>> | name1    | i-43-2147-VM  | Expunging |      2195 |  53687091200 | Expunged
    | 2014-10-16 08:12:04 | sata          |
>>>> | name2    | i-17-2123-VM  | Expunging |      2173 |  10737418240 | Expunged
    | 2014-10-15 19:14:30 | sata          |
>>>> | name3    | i-43-2052-VM  | Expunging |      2098 |  10737418240 | Expunged
    | 2014-10-07 11:57:49 | sata          |
>>>> | name4    | i-191-1954-VM | Expunging |      2000 |  42949672960 | Expunged
    | 2014-10-01 15:15:04 | sata          |
>>>> | name5    | i-132-1966-VM | Expunging |      2012 |  21474836480 | Expunged
    | 2014-09-29 20:33:12 | sata          |
>>>> | name6    | i-132-1967-VM | Expunging |      2013 |  21474836480 | Expunged
    | 2014-09-29 20:33:12 | sata          |
>>>> | name7    | i-132-1962-VM | Expunging |      2008 |  10737418240 | Expunged
    | 2014-09-29 20:33:11 | sata          |
>>>> | name8    | i-132-1964-VM | Expunging |      2010 |  10737418240 | Expunged
    | 2014-09-29 20:33:11 | sata          |
>>>>
>>>> Not all VMs with 'Expunged' state still present in VMware but a lot of from
them still weren't removed!
>>>>
>>>> mysql> select id,uuid,name,status,capacity_bytes,capacity_iops,tag,disk_used_capacity,disk_reserved_capacity
from storage_pool_view;
>>>> +----+--------------------------------------------------------------+---------------+-------------+----------------+---------------+------+--------------------+------------------------+
>>>> | id | uuid | name          | status      | capacity_bytes | capacity_iops
| tag  | disk_used_capacity | disk_reserved_capacity |
>>>> +----+--------------------------------------------------------------+---------------+-------------+----------------+---------------+------+--------------------+------------------------+
>>>> |  2 | 096b3b6e-481d-33b4-9b34-7178612a2535  |     SATA      | Up | 5497289703424
|          NULL | sata |      7913594810761 | 0 |
>>>> +----+--------------------------------------------------------------+---------------+-------------+----------------+---------------+------+--------------------+------------------------+
>>>> 5 rows in set (0.02 sec)
>>>>
>>>> Also I have found that Expunged volumes have 'chain_info' field:
>>>>
>>>> mysql> select name,removed,state,chain_info from volumes where id=2112
\G
>>>> *************************** 1. row ***************************
>>>>      name: ROOT-2066
>>>>   removed: 2014-10-09 11:57:52
>>>>     state: Expunged
>>>> chain_info: {"diskDeviceBusName":"scsi0:0","diskChain":["[SATA1] i-238-2066-VM/ROOT-2066.vmdk"]}
>>>> 1 row in set (0.00 sec)
>>>>
>>>>
>>>> BUT Destroyed VMs don't have it:
>>>> mysql> select name,removed,state,chain_info from volumes where id=2192
\G
>>>> *************************** 1. row ***************************
>>>>      name: ROOT-2143
>>>>   removed: 2014-10-15 18:42:48
>>>>     state: Destroy
>>>> chain_info: NULL
>>>> 1 row in set (0.00 sec)
>>>>
>>>>
>>>> So, the questions are:
>>>> - How CloudStack calculate allocated disk size for storage? Is it use disk
size from 'Expunged' volumes?
>>>> - Can I manually push CloudStack to remove 'Expunged' volumes from VMware?
>>>>
>>>> I haven't created bug in Jira but I can if it's needed.
>>>>
>>>> Thank you!
>>>>
>>>> -- 
>>>> Best regards,
>>>> Denis
>>>>
>>>
>>>
>>> ---
>>> This email is free from viruses and malware because avast! Antivirus protection
is active.
>>> http://www.avast.com
>>>
>>>
>>
>> -- 
>> Best regards,
>> Denis
>>
>
>
> ---
> This email is free from viruses and malware because avast! Antivirus protection is active.
> http://www.avast.com
>
>

-- 
Best regards,
Denis 


Mime
View raw message