cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ahmad Emneina <aemne...@gmail.com>
Subject Re: SystemVM offline causes extreme snapshot bloat
Date Thu, 07 Feb 2013 21:24:46 GMT
sounds painful. thanks for sharing this though. Hopefully someone can gain
value from this, and incorporate this into their operations.


On Thu, Feb 7, 2013 at 1:19 PM, Bryan Whitehead <driver@megahappy.net>wrote:

> This is on cloudstack 3.0.2. I had already removed all the snapshots in
> the cloudstack UI. So from the cloudstack point of view there was zero
> snapshots when in fact there was many more.
>
> The libvirt guys in #virt IRC were able to help me clean up all the
> internal snapshots on the qcow2 file via "virsh snapshot-delete [...]
> [...]". Unfortunately I cannot recover the 200G of space the qcow2 file has
> grown to without shutting down the VM since I'm running qemu/libvirt that
> is too old (CentOS 6.3 + some patches).
>
> According to libvirt there is 0 snapshots left but directly running
> "qemu-img info [qcow2 file]" shows there are still ~10 snapshots. (look at
> my original email - you can see the difference between "qemu-img info"
> output and "virsh snapshot-list"
>
> Anyhow, scheduled snapshots is working again just fine... including
> auto-rotation of a limit of 3-backups. But I will need to more closely
> monitor logs for systemvm storage problems... I can deal with a 220G qcow2
> file. Since many internal snapshots have been deleted there is a lot of
> "free" space in the qcow2 file - so hopefully it won't grow again if this
> happens again.
>
>
> On Tue, Feb 5, 2013 at 1:33 PM, Ahmad Emneina <aemneina@gmail.com> wrote:
>
>> Bryan* ... sorry
>>
>>
>> On Tue, Feb 5, 2013 at 1:33 PM, Ahmad Emneina <aemneina@gmail.com> wrote:
>>
>> > I'd also file a bug for this Brian. The process was failing, and not
>> > recovering gracefully, by cleaning up after itself.
>> >
>> >
>> > On Tue, Feb 5, 2013 at 1:18 PM, Ahmad Emneina <aemneina@gmail.com>
>> wrote:
>> >
>> >> I think in 4.0 and master this was broken. Marcus or Edison could
>> >> probably comment to that. Can you try deleting the snapshots from the
>> >> cloudstack UI? If that doesnt work, we can work to devise a method
>> where we
>> >> query the db, identify valid snapshots, then using virsh commands
>> delete
>> >> the failed snapshots.
>> >>
>> >>
>> >> On Fri, Feb 1, 2013 at 4:02 PM, Bryan Whitehead <driver@megahappy.net
>> >wrote:
>> >>
>> >>> For some reason my storageVM was in a state where things "worked" but
>> >>> automated snapshots were not working.
>> >>>
>> >>> Basically the step where libvirt created a stapshot on the primary
>> >>> storage
>> >>> worked - but the step to have the snapshot parent copied to the
>> secondary
>> >>> storage was not working. I got automated snapshots working again by
>> >>> rebooting the storage systemVM.
>> >>>
>> >>> However, when i look at libvirt I can see there are many snapshots on
>> the
>> >>> primary storage. A 12G filesystem has a qcow2 size of 220GB.
>> >>>
>> >>> virsh snapshot-list --parent i-3-14-VM
>> >>>  Name                 Creation Time             State           Parent
>> >>> ------------------------------------------------------------
>> >>>  24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130125025313
>> 2013-01-25
>> >>> 02:53:13 +0000 running
>> >>>  24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130126025313
>> 2013-01-26
>> >>> 02:53:13 +0000 running
>> >>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130125025313
>> >>>  24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130127025313
>> 2013-01-27
>> >>> 02:53:13 +0000 running
>> >>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130126025313
>> >>>  24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130128025313
>> 2013-01-28
>> >>> 02:53:13 +0000 running
>> >>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130127025313
>> >>>  24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130129025313
>> 2013-01-29
>> >>> 02:53:13 +0000 running
>> >>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130128025313
>> >>>  24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130130025313
>> 2013-01-30
>> >>> 02:53:13 +0000 running
>> >>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130129025313
>> >>>  24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130131025313
>> 2013-01-31
>> >>> 02:53:13 +0000 running
>> >>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130130025313
>> >>>  24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130201025313
>> 2013-02-01
>> >>> 02:53:13 +0000 running
>> >>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130131025313
>> >>>  24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130201223643
>> 2013-02-01
>> >>> 22:36:43 +0000 running
>> >>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130201025313
>> >>>
>> >>> The actual qcow2 file looks like a much bigger wreck:
>> >>>
>> >>> qemu-img info
>> /gluster/qcow2/images/7ec192c1-ecee-4f5e-8e3d-567f7878ceb1
>> >>> image: /gluster/qcow2/images/7ec192c1-ecee-4f5e-8e3d-567f7878ceb1
>> >>> file format: qcow2
>> >>> virtual size: 100G (107374182400 bytes)
>> >>> disk size: 211G
>> >>> cluster_size: 65536
>> >>> backing file:
>> /gluster/qcow2/images/cb151441-209c-4f43-a2a4-b390c9bb9768
>> >>> (actual path:
>> /gluster/qcow2/images/cb151441-209c-4f43-a2a4-b390c9bb9768)
>> >>> Snapshot list:
>> >>> ID        TAG                 VM SIZE                DATE       VM
>> CLOCK
>> >>> 1         24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20121230025415
>> >>> 1.0G 2012-12-30 02:54:16  785:39:01.680
>> >>> 2         24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20121231025415
>> >>> 1.0G 2012-12-31 02:54:16  809:35:35.417
>> >>> 3         24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130101025415
>> >>> 1.0G 2013-01-01 02:54:16  833:24:20.046
>> >>> 4         24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130102025415
>> >>> 1.0G 2013-01-02 02:54:15  857:13:02.766
>> >>> 5         24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130103025415
>> >>> 1.0G 2013-01-03 02:54:15  881:01:42.467
>> >>> 6         24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130104025415
>> >>> 1.0G 2013-01-04 02:54:15  904:50:29.840
>> >>> 7         24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130105025415
>> >>> 1.0G 2013-01-05 02:54:16  928:39:17.670
>> >>> 8         24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130106025415
>> >>> 1.0G 2013-01-06 02:54:16  952:28:07.139
>> >>> 9         24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130107025415
>> >>> 1.1G 2013-01-07 02:54:16  976:16:58.396
>> >>> 10        24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130108025415
>> >>> 1.1G 2013-01-08 02:54:15 1000:05:45.624
>> >>> 11        24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130109025415
>> >>> 1.1G 2013-01-09 02:54:15 1023:54:25.482
>> >>> 12        24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130110025415
>> >>> 1.1G 2013-01-10 02:54:16 1047:43:17.503
>> >>> 13        24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130111025415
>> >>> 1.1G 2013-01-11 02:54:16 1071:32:03.747
>> >>> 14        24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130112025415
>> >>> 1.1G 2013-01-12 02:54:16 1095:20:52.220
>> >>> 15        24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130113025415
>> >>> 1.1G 2013-01-13 02:54:16 1119:09:43.387
>> >>> 16        24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130114025415
>> >>> 1.1G 2013-01-14 02:54:16 1142:58:32.657
>> >>> 17        24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130115025415
>> >>> 1.1G 2013-01-15 02:54:16 1166:47:20.084
>> >>> 18        24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130116025416
>> >>> 1.1G 2013-01-16 02:54:16 1190:36:06.729
>> >>> 19        24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130117025416
>> >>> 1.1G 2013-01-17 02:54:16 1214:24:53.904
>> >>> 20        24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130118025415
>> >>> 1.1G 2013-01-18 02:54:16 1238:13:35.640
>> >>> 21        24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130119025415
>> >>> 1.1G 2013-01-19 02:54:16 1262:02:17.839
>> >>> 22        24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130120025415
>> >>> 1.1G 2013-01-20 02:54:16 1285:51:01.114
>> >>> 23        24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130121025415
>> >>> 1.1G 2013-01-21 02:54:16 1309:39:49.956
>> >>> 24        24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130122025416
>> >>> 1.1G 2013-01-22 02:54:16 1333:28:39.703
>> >>> 25        24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130123025415
>> >>> 1.1G 2013-01-23 02:54:16 1357:17:25.426
>> >>> 26        24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130124025416
>> >>> 1.1G 2013-01-24 02:54:16 1381:06:04.731
>> >>> 27        24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130125025313
>> >>> 517M 2013-01-25 02:53:13   04:46:52.307
>> >>> 28        24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130126025313
>> >>> 678M 2013-01-26 02:53:13   28:44:35.134
>> >>> 29        24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130127025313
>> >>> 918M 2013-01-27 02:53:13   52:42:00.249
>> >>> 30        24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130128025313
>> >>> 1.1G 2013-01-28 02:53:13   76:38:55.428
>> >>> 31        24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130129025313
>> >>> 1.2G 2013-01-29 02:53:13  100:35:27.978
>> >>> 32        24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130130025313
>> >>> 1.2G 2013-01-30 02:53:13  124:31:55.179
>> >>> 33        24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130131025313
>> >>> 1.3G 2013-01-31 02:53:13  148:28:17.744
>> >>> 34        24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130201025313
>> >>> 1.3G 2013-02-01 02:53:13  172:24:33.630
>> >>> 35        24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130201223643
>> >>> 1.3G 2013-02-01 22:36:43  192:04:17.498
>> >>>
>> >>> Suggestions on how I can clean up these extra snapshots?
>> >>>
>> >>> -Bryan
>> >>>
>> >>
>> >>
>> >
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message