cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From cloudstack-fan <cloudstack-...@protonmail.com.INVALID>
Subject qemu2 images are being corrupted
Date Mon, 02 Jul 2018 09:21:24 GMT
Dear colleagues,

I'm posting as an anonymous user, because there's a thing that concerns me a little and I'd
like to share my experience with you, so maybe some people could relate to the same. ACS is
amazing, it solves my tasks for 6 years, I'm running a few ACS-backed clouds that contain
hundreds and hundreds of VMs. I'm enjoying ACS really much, but there's a thing that scares
me sometimes.

It happens pretty seldom, but the more VMs you have is the more chances you run into this
glitch. It usually happens on the sly and you don't get any error messages in log-files of
your cloudstack-management server or a cloudstack-agent, so you don't even know that something
had happened until you see that a virtual machine is having major problems. If you're lucky,
you see it on the same day when it happens, but if you aren't - you won't suspect anything
unusual for a week, but at some moment you realize that the filesystem had become a mess and
you can't do anything to restore it. You're trying to restore it from a snapshot, but if you
don't have a snapshot that would be created before the incident, your snapshots won't help.
:-(

I experienced it for about 5-7 times during the last 5-6 years and there are a few conditions
that always present:
 * it happens on KVM-based hosts (I experienced itt with CentOS 6 and CentOS 7) with qcow2-images
(either 0.10 and 1.1 versions);
 * it happens on primary storages running different filesystems (I experiences it with local
XFS and network-based GFS2 and NFS);
 * it happens when a volume snapshot is being made, according to the log-files inside of a
VM (guest's operating system's kernel starts complaining on a filesystem errors);
 * at the same time, as I wrote before, there are NO error messages in the log-files outside
of a VM which disk image is corrupted;
 * but when you run `qemu-img check ...` to check the image, you may see a lot of leaked clusters
(that's why I'd strongly advice to check each and every image one each and every primary storage
at least once per hour by a script being run by your monitoring system, something kind of
`for imagefile in $(find /var/lib/libvirt/images -maxdepth 1 -type f); do { /usr/bin/qemu-img
check "${imagfile}"; if [[ ${?} -ne 0 ]]; then { ... } fi; } done`);
 * when it happens you can also find a record in the snapshot_store_ref table that refers
to the snapshot on a primary storage (see an example here https://pastebin.com/BuxCXVSq) -
this record should have been removed when the snapshot's state is being changed from "BackingUp"
to "BackedUp", but it isn't being removed in this case. At the same time, this snapshot isn't
being listed in the output of `qemu-img snapshot -l ...`, so that's why I suppose that the
image is being corrupted when ACS deletes the snapshot that has been backed up (it tries to
delete the snapshot, but something goes wrong, image is being corrupted, but ACS thinks that
everything's fine and changes the status to "BackedUp" without a bit of qualm);
 * if you're trying to restore this VM's image from the same snapshot that has caused destruction
or any other snapshot that has been made after that, you'll find the same corrupted filesystem
inside, but the snapshot's image that is stored in your secondary storage doesn't show anything
wrong when you run `qemu-img check ...` (so you can restore your image only if you have a
snapshot that had been created AND stored before the incident).

As I wrote, I saw several times in different environments and different versions of ACS. I'm
pretty sure that it's not only me who had such a luck to experience the same glitch, so let's
share our stories. Maybe together we'll find out why does it happen and how to prevent that
in future.

Thanks in advance,
An Anonymous ACS Fan
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message