Vladim, thank you again for follow up. The references below, found or not found are to CS MySql database. All the files are actually there, they are LV devices in CLVM, I just did not find any references in CS database for their UUIDs. The corrupt one is: /dev/VG_XenStorage-3b6e386d-8736-6b6b-7006-f3f4df9bd586/VHD-1a240d45-ee0a-4c30-809b-3114dfaf85ba I know that is corrupt, because there are lots of entries in SMlog for it like this: [8756] 2015-12-07 21:50:45.118758 ['/usr/sbin/vhd-util', 'check', '--debug', '-n', '/dev/VG_XenStorage-3b6e386d-8736-6b6b-7006-f3f4df9bd586/VHD-1a240d45-ee0a-4c30-809b-3114dfaf85ba'] [8756] 2015-12-07 21:50:45.131975 FAILED: (rc 22) stdout: 'primary footer invalid: invalid cookie /dev/VG_XenStorage-3b6e386d-8736-6b6b-7006-f3f4df9bd586/VHD-1a240d45-ee0a-4c30-809b-3114dfaf85ba appears invalid; dumping metadata VHD Footer Summary: ------------------- Cookie : conectix Features : (0x00000002) File format version : Major: 1, Minor: 0 Data offset : 512 Timestamp : Mon Dec 7 18:19:46 2015 Creator Application : 'tap' Creator version : Major: 1, Minor: 3 Creator OS : Unknown! Original disk size : 20480 MB (21474836480 Bytes) Current disk size : 20480 MB (21474836480 Bytes) Geometry : Cyl: 41610, Hds: 16, Sctrs: 63 : = 20479 MB (21474754560 Bytes) Disk type : Differencing hard disk Checksum : 0xffffef85|0xffffef85 (Good!) UUID : 8f25299d-d7cc-44fe-a42b-95b4e9a31a47 Saved state : No Hidden : 1 VHD Header Summary: ------------------- Cookie : cxsparse Data offset (unusd) : 18446744073709 Table offset : 1536 Header version : 0x00010000 Max BAT size : 1048576 Block size : 2097152 (2 MB) Parent name : VG_XenStorage--3b6e386d--8736--6b6b--7006--f3f4df9bd586-VHD--02e8b56b--279c--4d5b--8870--b3c5f2255dc7 Parent UUID : 8cece4c9-8c94-47b9-9f7a-cb6023da9b72 Parent timestamp : Mon Dec 7 18:19:46 2015 Checksum : 0xffffc6ed|0xffffc6ed (Good!) VHD Parent Locators: -------------------- locator: : 0 code : PLAT_CODE_MACX data_space : 512 data_length : 110 data_offset : 4327424 decoded name : ./VG_XenStorage--3b6e386d--8736--6b6b--7006--f3f4df9bd586-VHD--02e8b56b--279c--4d5b--8870--b3c5f2255dc7 locator: : 1 code : PLAT_CODE_W2KU data_space : 512 data_length : 206 data_offset : 4327936 decoded name : ./VG_XenStorage--3b6e386d--8736--6b6b--7006--f3f4df9bd586-VHD--02e8b56b--279c--4d5b--8870--b3c5f2255dc7 locator: : 2 code : PLAT_CODE_W2RU data_space : 512 data_length : 206 data_offset : 4328448 decoded name : ./VG_XenStorage--3b6e386d--8736--6b6b--7006--f3f4df9bd586-VHD--02e8b56b--279c--4d5b--8870--b3c5f2255dc7 VHD Batmap Summary: ------------------- Batmap offset : 4196352 Batmap size (secs) : 256 Batmap version : 0x00010002 Checksum : 0xffffff7f|0xffffff7f (Good!) ', stderr: '' [8756] 2015-12-07 21:50:45.132293 ['/usr/sbin/vhd-util', 'query', '--debug', '-s', '-n', '/dev/VG_XenStorage-3b6e386d-8736-6b6b-7006-f3f4df9bd586/VHD-1a240d45-ee0a-4c30-809b-3114dfaf85ba'] [8756] 2015-12-07 21:50:45.145222 SUCCESS [8756] 2015-12-07 21:50:45.145502 lock: tried lock /var/lock/sm/3b6e386d-8736-6b6b-7006-f3f4df9bd586/sr, acquired: True (exists: True) [8756] 2015-12-07 21:50:45.145635 lock: released /var/lock/sm/3b6e386d-8736-6b6b-7006-f3f4df9bd586/sr [8756] 2015-12-07 21:50:45.145752 ['/usr/sbin/lvchange', '/dev/VG_XenStorage-3b6e386d-8736-6b6b-7006-f3f4df9bd586/VHD-1a240d45-ee0a-4c30-809b-3114dfaf85ba', '-p', 'r'] .. .. .. <8756> 2015-12-07 21:50:45.229864 *~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~* <8756> 2015-12-07 21:50:45.229919 *********************** <8756> 2015-12-07 21:50:45.229969 * E X C E P T I O N * <8756> 2015-12-07 21:50:45.230017 *********************** <8756> 2015-12-07 21:50:45.230077 coalesce: EXCEPTION util.SMException, VHD *1a240d45[VHD](20.000G//136.000M|n) corrupted <8756> 2015-12-07 21:50:45.230129 File "/opt/xensource/sm/cleanup.py", line 1397, in coalesce self._coalesce(vdi) File "/opt/xensource/sm/cleanup.py", line 1587, in _coalesce vdi._doCoalesce() File "/opt/xensource/sm/cleanup.py", line 1050, in _doCoalesce self.parent.validate() File "/opt/xensource/sm/cleanup.py", line 1043, in validate VDI.validate(self, fast) File "/opt/xensource/sm/cleanup.py", line 633, in validate raise util.SMException("VHD %s corrupted" % self) <8756> 2015-12-07 21:50:45.230181 *~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~* <8756> 2015-12-07 21:50:45.230234 Coalesce failed, skipping Regards, F. On 10 Dec 2015, at 08:22, Vadim Kimlaychuk wrote: > Dear France, > > It is more than 6 months I have used vhd-util, and xe to clean-up manually, but I remember that if you try to destroy VDI that has a child - you'll get an error. Since you have all the chain from the parent you should start from the last child and move to the top. You shouldn't be able to destroy VDI in the middle of the chain, since it will definitely corrupt data. > > I didn't understand how exactly did you check existing VHD files? How do you know it is corrupted? Does vhd-util query shows that? What does it mean - "corrupted and not found". If it is not found, how do you know it is corrupted? > > Try to work with vhd-util -- it is great. I think you can resolve all the issues with it. > > Vadim. > > > On 2015-12-09 20:39, France wrote: > >> Thank you Vladim, for your response. >> Ity is greatly appreciated. >> I will surely read the document. >> So If I understand correctly, I can try to remove it and if it would be bad for my setup, xe will not do it? :-) >> I guess coalesce will not work, if one VHD in chain is corrupted. >> I guess I will have to try and repair it first. >> We will see. :-) >> I vageuly remember that thin provisioning was not possible with CLVM over ISCSI at the time of cluster setup, so I guess we do not have it. >> Regards, >> F. >> On 09 Dec 2015, at 18:18, Vadim Kimlaychuk wrote: >> Dear France, >> Hope this article helps you:http://support.citrix.com/filedownload/CTX122978/XenServer_Understanding_Snapshots.pdf [1] Common practice is to use "vhd-util" to merge (coalesce) VHD files into usable image. Luckily xe tool is smart enough not to allow you destroy image that is a part of the chain. Of course it is more safe to copy all the files before destruction (in a case you have thin provisioning). >> Vadim. >> On 2015-12-09 18:27, France wrote: >> Hi guys, >> below is the chain cross referenced with CS MySql database. >> Can I delete/destroy 1a240d45-ee0a-4c30-809b-3114dfaf85ba without bad consequences? Are next in chain dependant upon it? >> *555b38bf[VHD](20.000G//2.695G|n) -> not foud >> 5afd5849[VHD](20.000G//8.000M|n) -> found in template_spool_ref. status ready, template ID 292 >> *02e8b56b[VHD](20.000G//12.000M|n) -> not found >> *1a240d45[VHD](20.000G//136.000M|n) <- corrupted and not found >> *c99e711b[VHD](20.000G//136.000M|n) -> not found >> 34a03c9a[VHD](20.000G//8.000M|a) -> found in shapshot_store_ref. state destroyed, volume id 661, from template id 292 >> 5c12db0a[VHD](20.000G//8.000M|n) -> found in shapshot_store_ref. state destroyed -> probably not part of the above chain? >> I would do xe vdi-destroy=1a240d45-ee0a-4c30-809b-3114dfaf85ba . >> Alternatively I think about exporting template ID 292 in CS, deleting it and importing it back again. It should remove the whole chain, right? 'Cause noone is using it? >> Regards, >> F. > > > > Links: > ------ > [1] http://support.citrix.com/filedownload/CTX122978/XenServer_Understanding_Snapshots.pdf