cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeremy Peterson <jpeter...@acentek.net>
Subject RE: Corrupted HDD
Date Mon, 25 Sep 2017 16:55:15 GMT
[root@Flex-Xen5 VG_XenStorage-469b6dcd-8466-3d03-de0e-cc3983e1b6e2]# xe vdi-list | grep -A
5 12eb0cdaba30
[root@Flex-Xen5 VG_XenStorage-469b6dcd-8466-3d03-de0e-cc3983e1b6e2]# xe vdi-list | grep -A
5 d750a907a699

You can see above that both uuid's I pulled one from the uuid not found command from cloudstack
and also from the sql query both are not showing anything when I do a xe vdi-list.  This tells
me that xenserver cannot find the vdi's  leaving me with only my block device.

[root@Flex-Xen5 mapper]# ls -lh | grep 699
brw-rw---- 1 root disk 253,   9 Sep 23 11:34 VG_XenStorage--469b6dcd--8466--3d03--de0e--cc3983e1b6e2-VHD--f5aabea5--2a18--4f24--9f8d--d750a907a699

Sorry I am responding so many times I'm just going through this one by one trying to figure
anything out to recover this data.  

Jeremy


-----Original Message-----
From: Jeremy Peterson [mailto:jpeterson@acentek.net] 
Sent: Monday, September 25, 2017 11:44 AM
To: users@cloudstack.apache.org
Subject: RE: Corrupted HDD

Now if I look at my starting error from cloudstack I see the following for when it calls the
ROOT disk.

"disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{
"uuid":"e772d773-5a9d-4c7b-9227-12eb0cdaba30",
"volumeType":"ROOT",



2017-09-25 11:20:16,555 DEBUG [c.c.a.t.Request] (Work-Job-Executor-57:ctx-dcbad35d job-261776/job-261778
ctx-f1843e45) Seq 19-8416664753602063711: Executing:  { Cmd , MgmtId:                    
          345050411715, via: 19(Flex-Xen1.flexhost.local), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StartCommand":{"vm":{"id":223,"name":"i-23-223-VM","bootloader":"PyGrub","typ
                             e":"User","cpus":2,"minSpeed":500,"maxSpeed":2000,"minRam":1073741824,"maxRam":4294967296,"arch":"x86_64","os":"Windows
Server 2012 R2 (64-bit)","platformEmulator":"Windows S                              erver
2012 R2 (64-bit)","bootArgs":"","enableHA":true,"limitCpuUse":true,"enableDynamicallyScaleVm":true,"vncPassword":"P2LDsJVeaBNz2+W4OB93ZA==","params":{"memoryOvercommitR
                             atio":"4","platform":"viridian:true;acpi:1;apic:true;viridian_reference_tsc:true;viridian_time_ref_count:true;pae:true;videoram:8;device_id:0002;nx:true;timeoffset:-17885;vga
                             :std","keyboard":"us","Message.ReservedCapacityFreed.Flag":"false","cpuOvercommitRatio":"4","hypervisortoolsversion":"xenserver61"},"uuid":"6ff4beb0-e064-4364-94d3-5b3fbc15ee
                             16","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"e772d773-5a9d-4c7b-9227-12eb0cdaba30","volumeType":"ROOT","dataStore":{"org.apache.cloudstack
                             .storage.to.PrimaryDataStoreTO":{"uuid":"FlexSAN2-LUN1","id":6,"poolType":"PreSetup","host":"localhost","path":"/FlexSAN2-LUN1","port":0,"url":"PreSetup://localhost/FlexSAN2-
                             LUN1/?ROLE=Primary&STOREUUID=FlexSAN2-LUN1"}},"name":"ROOT-223","size":171798691840,"path":"f5aabea5-2a18-4f24-9f8d-d750a907a699","volumeId":275,"vmName":"i-23-223-VM","accou
                             ntId":23,"format":"VHD","provisioningType":"THIN","id":275,"deviceId":0,"hypervisorType":"XenServer"}},"diskSeq":0,"path":"f5aabea5-2a18-4f24-9f8d-d750a907a699","type":"ROOT"
                             ,"_details":{"managed":"false","storagePort":"0","storageHost":"localhost","volumeSize":"171798691840"}},{"data":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"id":0,
                             "format":"ISO","accountId":0,"hvm":false}},"diskSeq":3,"type":"ISO"}],"nics":[{"deviceId":0,"networkRateMbps":1000,"defaultNic":true,"pxeDisable":false,"nicUuid":"d7d464bc-64
                             c1-490a-944e-6a2adee1a451","uuid":"940feda7-65ee-438b-bd06-d33ba3d734c4","ip":"10.1.1.126","netmask":"255.255.255.0","gateway":"10.1.1.1","mac":"02:00:62:78:00:06","dns1":"20
                             8.74.240.5","dns2":"208.74.247.245","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://1679","isolationUri":"vlan://1679","isSecurityGroupEnabled":false,"name":"GUE
                             ST-PUB"}],"vcpuMaxLimit":16},"hostIp":"10.90.2.111","executeInSequence":false,"wait":0}}]
}
2017-09-25 11:20:16,555 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-378:ctx-d30ac189)
Seq 19-8416664753602063711: Executing request
2017-09-25 11:20:16,560 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-378:ctx-d30ac189)
1. The VM i-23-223-VM is in Starting state.
2017-09-25 11:20:16,579 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-378:ctx-d30ac189)
Created VM a6739bcc-968f-abe6-d85c-a016de8b0b9f for i-23-223-VM
2017-09-25 11:20:16,585 WARN  [c.c.h.x.r.CitrixResourceBase] (DirectAgent-378:ctx-d30ac189)
Catch Exception: class com.xensource.xenapi.Types$UuidInvalid due to The uuid you        
                      supplied was invalid.
The uuid you supplied was invalid.

So in my mind I should see that uuid in /dev/mapper/ but I do not.

So I looked at Dag's post and wanted to pull down some info from sql.

+-------+-----------------------------------------+----------------+----------------------+----------+------------------+------------+--------------------+--------------+---------+-------+-------+----------------+--------------+--------------------------------------+----------+----------+
| vmid  | vmname                                  | vminstname     | vmdispname          
| vmacctid | vmacctname       | vmdomainid | vmdomname          | vmofferingid | vmspeed |
vmmem | volid | volname        | volsize      | volpath                              | voltype
 | volstate |
+-------+-----------------------------------------+----------------+----------------------+----------+------------------+------------+--------------------+--------------+---------+-------+-------+----------------+--------------+--------------------------------------+----------+----------+
|   223 | DC2FCE                                  | i-23-223-VM    | DC2FCE              
|       23 | admin            |         18 | fce.coop           |           29 |    2000 |
 4096 |   275 | ROOT-223       | 171798691840 | f5aabea5-2a18-4f24-9f8d-d750a907a699 | ROOT
    | Ready    |

This shows me that the volpath is *699

Now that matches up with the only VHD symbolic link I can find back to /dev/mapper/VG_XenStorage--

VG_XenStorage-469b6dcd-8466-3d03-de0e-cc3983e1b6e2]# ls *699
VHD-f5aabea5-2a18-4f24-9f8d-d750a907a699

I think the parent volume got deleted and I'm left with only the 699 volume seen above.

Do you have any suggestions on recovering from this?

Jeremy


-----Original Message-----
From: Jeremy Peterson [mailto:jpeterson@acentek.net] 
Sent: Monday, September 25, 2017 10:15 AM
To: users@cloudstack.apache.org
Subject: RE: Corrupted HDD

Ok so I will be going through both of the items listed.

But my problem is our xenserver is with iSCSI SAN's so we do not have any VHD files.

# ls VG_XenStorage--469b6dcd--8466--3d03--de0e--cc3983e1b6e2-VHD--f5aabea5--2a18--4f24--9f8d--d750a907a699
VG_XenStorage--469b6dcd--8466--3d03--de0e--cc3983e1b6e2-VHD--f5aabea5--2a18--4f24--9f8d--d750a907a699

Now I've tried to run a dd on that file to export and try to recover from that its corrupted.
 

I'm grasping here and will be looking at more information.  I need to recover data from this
VG_XenStorage file.

Thank you in advance.  Just wanted to let you know where I am sitting.

Jeremy


-----Original Message-----
From: Dag Sonstebo [mailto:Dag.Sonstebo@shapeblue.com] 
Sent: Monday, September 25, 2017 3:16 AM
To: users@cloudstack.apache.org
Subject: Re: Corrupted HDD

Hi Jeremy,

In addition to Adrian’s walk-through – take a look at the following blog article from
last year – it should give you some more options to try - http://www.shapeblue.com/recovery-of-vms-to-new-cloudstack-instance/


Regards,
Dag Sonstebo
Cloud Architect
ShapeBlue

On 25/09/2017, 03:49, "Adrian Sender" <asender@testlabs.com.au> wrote:

    Hi Jeremy,
    
    Some time ago I worked with Citrix in creating a article to mount and recover
    LVMoHBA.
    
    How to Mount Linux LVM Partition in a XenServer Host
    
    http://support.citrix.com/article/CTX117791
    
    The above CTX article only applies to raw LVM LV. In our case the VDIs are LVs
    with VHD headers in it. We'll have to use blktap to create tapdisks to open
    the VHD LV before we can create device maps for the partitions / LVs within
    the VDI.
    
    Up to activating the VG, the article still applies. After activating the
    VG/LVs, we'll need to manually create tapdisk to open the VHD LV before using
    kpartx to create maps from partition tables.
    
    NOTE: I've included output from my test environment for your reference
    
    1. Create tapdisk
    
    # tap-ctl allocate /dev/xen/blktap-2/tapdev0
    
    2. Spawn a new process
    
    # tap-ctl spawn
    
    tapdisk spawned with pid 30383
    
    # tap-ctl list
    30383 - - - -
    - 0 - - -
    
    4. Attach the tapdisk to the process
    
    tap-ctl attach -p PID -m minor
    
    # tap-ctl attach -p 30383 -m 0
    # tap-ctl list
    30383 0 0 - -
    
    
    5. Open the VHD LV
    
    Note you may need to activate the LVM first with lvchange -ay
    
    # tap-ctl open -p 30383 -m 0 -a
    vhd:/dev/VG_XenStorage-b1415cf5-07e2-14fa-cbf8-6490751da242/VHD-182ce043-bf98-45d9-926b-595bbd82ed8a
    
    6. Verify
    
    # tap-ctl list 
    30383 0 0 vhd
    /dev/VG_XenStorage-b1415cf5-07e2-14fa-cbf8-6490751da242/VHD-182ce043-bf98-45d9-926b-595bbd82ed8a
    
    7. Create device maps from partition tables using kpartx
    
    kpartx -av /dev/xen/blktap-2/tapdev0
    
    # kpartx -av /dev/xen/blktap-2/tapdev0
    add map tapdev0p1 (252:5): 0 1024000 linear /dev/xen/blktap-2/tapdev0 2048
    add map tapdev0p2 (252:6): 0 32872448 linear /dev/xen/blktap-2/tapdev0 1026048
    
    NOTE: in this case, it is a CentOS 7 VDI, contains a /boot partition and VG -
    linux and 2 LVs (root, home).
    
    Device maps were created in
    /dev/mapper/tapdev0pN
    
    8. Mount 1st partition
    
    # mount /dev/mapper/tapdev0p1 /mnt/boot
    
    9. Activate VG
    
    # vgchange -a y
    2 logical volume(s) in volume group "linux" now active
    
    # lvscan
    ACTIVE '/dev/linux/root' [13.67 GB] inherit
    ACTIVE '/dev/linux/home' [2.00 GB] inherit
    
    
    10. Mount root and home LVs
    
    mount -t xfs /dev/linux/root /mnt/root
    
    NOTE: XenServer 6.2 kernel does NOT support XFS
    
    mount -t ext4 /dev/linux/home /mnt/home
    
    11. After backing up data, reverse the procedure to release resources
    
    List partition mappings
    
    # kpartx -lv /dev/xen/blktap-2/tapdev0
    tapdev0p1 : 0 1024000 /dev/xen/blktap-2/tapdev0 2048
    tapdev0p2 : 0 32872448 /dev/xen/blktap-2/tapdev0 1026048
    
    De-active the VG (all LVs)
    
    # vgchange -an linux
    0 logical volume(s) in volume group "linux" now active
    
    Delete partition mappings
    
    # kpartx -dv /dev/xen/blktap-2/tapdev0
    del devmap : tapdev0p2
    del devmap : tapdev0p1
    
    12. Release the blktap resources
    
    # tap-ctl list
    30383 0 0 vhd
    /dev/VG_XenStorage-b1415cf5-07e2-14fa-cbf8-6490751da242/VHD-182ce043-bf98-45d9-926b-595bbd82ed8a
    
    # tap-ctl close -p 30383 -m 0
    [root@xenserver-zilblqhl /]# tap-ctl list
    30383 0 0x2 - -
    
    # tap-ctl destroy -p 30383 -m 0
    [root@xenserver-zilblqhl /]# tap-ctl list
    
    Regards,
    Adrian Sender
    
    
    
    -----
Dag.Sonstebo@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK @shapeblue
  
 

----- Original Message -----------
    From: Jeremy Peterson <jpeterson@acentek.net>
    To: "users@cloudstack.apache.org" <users@cloudstack.apache.org>
    Sent: Sun, 24 Sep 2017 22:38:29 +0000
    Subject: Corrupted HDD
    
    > So I need to know if anyone can drop a hint on this.
    > 
    > How can i recover data from cloudstack with xenserver.  One of the 
    > storage LUN's got 100% full.  I was carefully deleting un-associated 
    > disks from the LUN.  A disk that had no VM attached.  Then when one 
    > of the VM's rebooted it came online with no bootable disk found.
    > 
    > Now I'm thinking this over that one of those disks were the a master 
    > UUID and thats why my VM's wont boot.  Can that been fixed?
    > 
    > I look in primary storage and can find the UUID of a disk matches a 
    > volume in /dev/mapper/VG.....
    > 
    > Can I pull that disk down and create a VHD to deploy a template and 
    > deploy a new vm from that vhd file?
    > 
    > Can I dd that disk to export to anything to try and recover data?  
    > Office files or PDF's are critical.
    > 
    > Does anyone know if there are any good consulting companies with 
    > Citrix or if Citrix can help with this?
    > 
    > Jeremy
    ------- End of Original Message -------
    
    

Mime
View raw message