cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dag Sonstebo <Dag.Sonst...@shapeblue.com>
Subject Re: Help me locate the VM owner of (corrupt) VDI - also 'primary footer invalid: invalid cookie' error
Date Tue, 08 Dec 2015 08:46:58 GMT
Hi France,

have a look at the following writeup I did a few weeks back - http://www.shapeblue.com/recovery-of-vms-to-new-cloudstack-instance/

It should give you an idea how to troubleshoot the VHD chain and how to tie it back to DB
entries and potential VM owner.


Regards,

Dag Sonstebo
Cloud Architect
ShapeBlue





On 07/12/2015, 22:14, "France" <mailinglists@isg.si> wrote:

>Hi Guys,
>
>after battling with cloudstack (4.3.2+) for the whole day with another issue (unable to
create snapshot’s for one VM, which I also failed to fix) I ran into message on XenServer
6.0.2+Hotfixes SMlog (copy pasted at the bottom).
>
>Even thou i kinda understand XenServers relations of VM -> VBD -> VDI -> SR ->
PBD, i failed to find out to which VM does this corrupted .vhd snapshot belong.
>Can you please help me find it? Afterwards, we can find the appropriate action in order
to remove the error and make coalesce work.
>
>The physical file with corruption is:
>/dev/VG_XenStorage-3b6e386d-8736-6b6b-7006-f3f4df9bd586/VHD-1a240d45-ee0a-4c30-809b-3114dfaf85ba
>
>This is the VDI it supposedly belongs to:
>[root@x3 log]# xe vdi-list uuid=02e8b56b-279c-4d5b-8870-b3c5f2255dc7 params=all
>uuid ( RO)                    : 02e8b56b-279c-4d5b-8870-b3c5f2255dc7
>              name-label ( RW): base copy
>        name-description ( RW):
>           is-a-snapshot ( RO): false
>             snapshot-of ( RO): <not in database>
>               snapshots ( RO):
>           snapshot-time ( RO): 19700101T00:00:00Z
>      allowed-operations (SRO): generate_config; update; resize; destroy; clone; copy;
snapshot
>      current-operations (SRO):
>                 sr-uuid ( RO): 3b6e386d-8736-6b6b-7006-f3f4df9bd586
>           sr-name-label ( RO): 1c8dbf77-2b23-3244-91ee-5037cb2a55a8
>               vbd-uuids (SRO):
>         crashdump-uuids (SRO):
>            virtual-size ( RO): 21474836480
>    physical-utilisation ( RO): 12582912
>                location ( RO): 02e8b56b-279c-4d5b-8870-b3c5f2255dc7
>                    type ( RO): User
>                sharable ( RO): false
>               read-only ( RO): true
>            storage-lock ( RO): false
>                 managed ( RO): false
>                  parent ( RO): <not in database>
>                 missing ( RO): false
>            other-config (MRW):
>           xenstore-data (MRO):
>               sm-config (MRO): vhd-parent: 555b38bf-fe63-41b2-9609-e16f4e26b274; vdi_type:
vhd; vhd-blocks: eJxjYBgFo2AUjIKRCQAFAAAB
>                 on-boot ( RW): persist
>           allow-caching ( RW): false
>         metadata-latest ( RO): false
>        metadata-of-pool ( RO): <not in database>
>                    tags (SRW):
>
>(It supposedly resides on this SR:
>uuid ( RO)                : 3b6e386d-8736-6b6b-7006-f3f4df9bd586
>          name-label ( RW): 1c8dbf77-2b23-3244-91ee-5037cb2a55a8
>    name-description ( RW): 208 - censored
>                host ( RO): <shared>
>                type ( RO): lvmoiscsi
>        content-type ( RO): user
>
>Which is on one of many HA iscsi targets mounted on the XenServer cluster.
>uuid ( RO)                  : f471ef16-4670-2f5e-967e-7e98d4721fe7
>             host-uuid ( RO): f6d699aa-f58f-449e-9a51-6e965562f178
>               sr-uuid ( RO): 3b6e386d-8736-6b6b-7006-f3f4df9bd586
>         device-config (MRO): targetIQN: censoredi:storage.censored; target: censored;
SCSIid: 1target2lun1
>    currently-attached ( RO): true)
>
>Cleaned exempt from SMlog on cluster master:
>
>[8756] 2015-12-07 21:50:45.118758      ['/usr/sbin/vhd-util', 'check', '--debug', '-n',
'/dev/VG_XenStorage-3b6e386d-8736-6b6b-7006-f3f4df9bd586/VHD-1a240d45-ee0a-4c30-809b-3114dfaf85ba']
>[8756] 2015-12-07 21:50:45.131975      FAILED: (rc 22) stdout: 'primary footer invalid:
invalid cookie
>/dev/VG_XenStorage-3b6e386d-8736-6b6b-7006-f3f4df9bd586/VHD-1a240d45-ee0a-4c30-809b-3114dfaf85ba
appears invalid; dumping metadata
>VHD Footer Summary:
>-------------------
>Cookie              : conectix
>Features            : (0x00000002) <RESV>
>File format version : Major: 1, Minor: 0
>Data offset         : 512
>Timestamp           : Mon Dec  7 18:19:46 2015
>Creator Application : 'tap'
>Creator version     : Major: 1, Minor: 3
>Creator OS          : Unknown!
>Original disk size  : 20480 MB (21474836480 Bytes)
>Current disk size   : 20480 MB (21474836480 Bytes)
>Geometry            : Cyl: 41610, Hds: 16, Sctrs: 63
>                    : = 20479 MB (21474754560 Bytes)
>Disk type           : Differencing hard disk
>Checksum            : 0xffffef85|0xffffef85 (Good!)
>UUID                : 8f25299d-d7cc-44fe-a42b-95b4e9a31a47
>Saved state         : No
>Hidden              : 1
>
>VHD Header Summary:
>-------------------
>Cookie              : cxsparse
>Data offset (unusd) : 18446744073709
>Table offset        : 1536
>Header version      : 0x00010000
>Max BAT size        : 1048576
>Block size          : 2097152 (2 MB)
>Parent name         : VG_XenStorage--3b6e386d--8736--6b6b--7006--f3f4df9bd586-VHD--02e8b56b--279c--4d5b--8870--b3c5f2255dc7
>Parent UUID         : 8cece4c9-8c94-47b9-9f7a-cb6023da9b72
>Parent timestamp    : Mon Dec  7 18:19:46 2015
>Checksum            : 0xffffc6ed|0xffffc6ed (Good!)
>
>VHD Parent Locators:
>--------------------
>locator:            : 0
>       code         : PLAT_CODE_MACX
>       data_space   : 512
>       data_length  : 110
>       data_offset  : 4327424
>       decoded name : ./VG_XenStorage--3b6e386d--8736--6b6b--7006--f3f4df9bd586-VHD--02e8b56b--279c--4d5b--8870--b3c5f2255dc7
>
>locator:            : 1
>       code         : PLAT_CODE_W2KU
>       data_space   : 512
>       data_length  : 206
>       data_offset  : 4327936
>       decoded name : ./VG_XenStorage--3b6e386d--8736--6b6b--7006--f3f4df9bd586-VHD--02e8b56b--279c--4d5b--8870--b3c5f2255dc7
>
>locator:            : 2
>       code         : PLAT_CODE_W2RU
>       data_space   : 512
>       data_length  : 206
>       data_offset  : 4328448
>       decoded name : ./VG_XenStorage--3b6e386d--8736--6b6b--7006--f3f4df9bd586-VHD--02e8b56b--279c--4d5b--8870--b3c5f2255dc7
>
>VHD Batmap Summary:
>-------------------
>Batmap offset       : 4196352
>Batmap size (secs)  : 256
>Batmap version      : 0x00010002
>Checksum            : 0xffffff7f|0xffffff7f (Good!)
>
>', stderr: ''
>[8756] 2015-12-07 21:50:45.132293      ['/usr/sbin/vhd-util', 'query', '--debug', '-s',
'-n', '/dev/VG_XenStorage-3b6e386d-8736-6b6b-7006-f3f4df9bd586/VHD-1a240d45-ee0a-4c30-809b-3114dfaf85ba']
>[8756] 2015-12-07 21:50:45.145222      SUCCESS
>[8756] 2015-12-07 21:50:45.145502      lock: tried lock /var/lock/sm/3b6e386d-8736-6b6b-7006-f3f4df9bd586/sr,
acquired: True (exists: True)
>[8756] 2015-12-07 21:50:45.145635      lock: released /var/lock/sm/3b6e386d-8736-6b6b-7006-f3f4df9bd586/sr
>[8756] 2015-12-07 21:50:45.145752      ['/usr/sbin/lvchange', '/dev/VG_XenStorage-3b6e386d-8736-6b6b-7006-f3f4df9bd586/VHD-1a240d45-ee0a-4c30-809b-3114dfaf85ba',
'-p', 'r']
><8756> 2015-12-07 21:50:45.229864      *~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*
><8756> 2015-12-07 21:50:45.229919               ***********************
><8756> 2015-12-07 21:50:45.229969               *  E X C E P T I O N  *
><8756> 2015-12-07 21:50:45.230017               ***********************
><8756> 2015-12-07 21:50:45.230077      coalesce: EXCEPTION util.SMException, VHD
*1a240d45[VHD](20.000G//136.000M|n) corrupted
><8756> 2015-12-07 21:50:45.230129        File "/opt/xensource/sm/cleanup.py", line
1397, in coalesce
>    self._coalesce(vdi)
>  File "/opt/xensource/sm/cleanup.py", line 1587, in _coalesce
>    vdi._doCoalesce()
>  File "/opt/xensource/sm/cleanup.py", line 1050, in _doCoalesce
>    self.parent.validate()
>  File "/opt/xensource/sm/cleanup.py", line 1043, in validate
>    VDI.validate(self, fast)
>  File "/opt/xensource/sm/cleanup.py", line 633, in validate
>    raise util.SMException("VHD %s corrupted" % self)
>
><8756> 2015-12-07 21:50:45.230181      *~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*
><8756> 2015-12-07 21:50:45.230234      Coalesce failed, skipping
>
>
>Sorry for possible mistakes, but after working for 13 hours, my concentration has dropped
and thank you for your help.
>
>Regards,
>F.
Find out more about ShapeBlue and our range of CloudStack related services

IaaS Cloud Design & Build<http://shapeblue.com/iaas-cloud-design-and-build//>
CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/>
CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/>
CloudStack Software Engineering<http://shapeblue.com/cloudstack-software-engineering/>
CloudStack Infrastructure Support<http://shapeblue.com/cloudstack-infrastructure-support/>
CloudStack Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>

This email and any attachments to it may be confidential and are intended solely for the use
of the individual to whom it is addressed. Any views or opinions expressed are solely those
of the author and do not necessarily represent those of Shape Blue Ltd or related companies.
If you are not the intended recipient of this email, you must neither take any action based
upon its contents, nor copy or show it to anyone. Please contact the sender if you believe
you have received this email in error. Shape Blue Ltd is a company incorporated in England
& Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated
under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated
in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company
registered by The Republic of South Africa and is traded under license from Shape Blue Ltd.
ShapeBlue is a registered trademark.
Mime
View raw message