cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dag Sonstebo <Dag.Sonst...@shapeblue.com>
Subject Re: Corrupted HDD
Date Tue, 26 Sep 2017 08:25:46 GMT
Hi Jeremy,

If you revisit the info on vhd-util this will still help you troubleshoot LVM based storage.
Another KB article you can refer to is https://support.citrix.com/article/CTX201296 - as you
can see from this the scans will work for both (e.g. SIC #vhd-util scan –m “VHD-*” -f
-c -l VG_XenStorage-<SR-UUID> -p –v}

In short though – if your base disk is gone I’m not aware of any mechanism to recover
data from your delta disk.

Regards,
Dag Sonstebo
Cloud Architect
ShapeBlue

On 25/09/2017, 18:02, "Jeremy Peterson" <jpeterson@acentek.net> wrote:

    Now looking at Adrian's link 
    
    Since the VM is powered off and will not boot due to UUID mismatch.
    
    [root@Flex-Xen5 mapper]# xe vm-list name-label=i-23-223-VM
    
    I don't get anything after issuing the vm-list command for that VM.
    
    Jeremy
    
    
    -----Original Message-----
    From: Jeremy Peterson [mailto:jpeterson@acentek.net] 
    Sent: Monday, September 25, 2017 11:55 AM
    To: users@cloudstack.apache.org
    Subject: RE: Corrupted HDD
    
    [root@Flex-Xen5 VG_XenStorage-469b6dcd-8466-3d03-de0e-cc3983e1b6e2]# xe vdi-list | grep
-A 5 12eb0cdaba30
    [root@Flex-Xen5 VG_XenStorage-469b6dcd-8466-3d03-de0e-cc3983e1b6e2]# xe vdi-list | grep
-A 5 d750a907a699
    
    You can see above that both uuid's I pulled one from the uuid not found command from cloudstack
and also from the sql query both are not showing anything when I do a xe vdi-list.  This tells
me that xenserver cannot find the vdi's  leaving me with only my block device.
    
    [root@Flex-Xen5 mapper]# ls -lh | grep 699
    brw-rw---- 1 root disk 253,   9 Sep 23 11:34 VG_XenStorage--469b6dcd--8466--3d03--de0e--cc3983e1b6e2-VHD--f5aabea5--2a18--4f24--9f8d--d750a907a699
    
    Sorry I am responding so many times I'm just going through this one by one trying to figure
anything out to recover this data.  
    
    Jeremy
    
    
    -----Original Message-----
    From: Jeremy Peterson [mailto:jpeterson@acentek.net] 
    Sent: Monday, September 25, 2017 11:44 AM
    To: users@cloudstack.apache.org
    Subject: RE: Corrupted HDD
    
    Now if I look at my starting error from cloudstack I see the following for when it calls
the ROOT disk.
    
    "disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{
    "uuid":"e772d773-5a9d-4c7b-9227-12eb0cdaba30",
    "volumeType":"ROOT",
    
    
    
    2017-09-25 11:20:16,555 DEBUG [c.c.a.t.Request] (Work-Job-Executor-57:ctx-dcbad35d job-261776/job-261778
ctx-f1843e45) Seq 19-8416664753602063711: Executing:  { Cmd , MgmtId:                    
          345050411715, via: 19(Flex-Xen1.flexhost.local), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StartCommand":{"vm":{"id":223,"name":"i-23-223-VM","bootloader":"PyGrub","typ
                             e":"User","cpus":2,"minSpeed":500,"maxSpeed":2000,"minRam":1073741824,"maxRam":4294967296,"arch":"x86_64","os":"Windows
Server 2012 R2 (64-bit)","platformEmulator":"Windows S                              erver
2012 R2 (64-bit)","bootArgs":"","enableHA":true,"limitCpuUse":true,"enableDynamicallyScaleVm":true,"vncPassword":"P2LDsJVeaBNz2+W4OB93ZA==","params":{"memoryOvercommitR
                             atio":"4","platform":"viridian:true;acpi:1;apic:true;viridian_reference_tsc:true;viridian_time_ref_count:true;pae:true;videoram:8;device_id:0002;nx:true;timeoffset:-17885;vga
                             :std","keyboard":"us","Message.ReservedCapacityFreed.Flag":"false","cpuOvercommitRatio":"4","hypervisortoolsversion":"xenserver61"},"uuid":"6ff4beb0-e064-4364-94d3-5b3fbc15ee
                             16","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"e772d773-5a9d-4c7b-9227-12eb0cdaba30","volumeType":"ROOT","dataStore":{"org.apache.cloudstack
                             .storage.to.PrimaryDataStoreTO":{"uuid":"FlexSAN2-LUN1","id":6,"poolType":"PreSetup","host":"localhost","path":"/FlexSAN2-LUN1","port":0,"url":"PreSetup://localhost/FlexSAN2-
                             LUN1/?ROLE=Primary&STOREUUID=FlexSAN2-LUN1"}},"name":"ROOT-223","size":171798691840,"path":"f5aabea5-2a18-4f24-9f8d-d750a907a699","volumeId":275,"vmName":"i-23-223-VM","accou
                             ntId":23,"format":"VHD","provisioningType":"THIN","id":275,"deviceId":0,"hypervisorType":"XenServer"}},"diskSeq":0,"path":"f5aabea5-2a18-4f24-9f8d-d750a907a699","type":"ROOT"
                             ,"_details":{"managed":"false","storagePort":"0","storageHost":"localhost","volumeSize":"171798691840"}},{"data":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"id":0,
                             "format":"ISO","accountId":0,"hvm":false}},"diskSeq":3,"type":"ISO"}],"nics":[{"deviceId":0,"networkRateMbps":1000,"defaultNic":true,"pxeDisable":false,"nicUuid":"d7d464bc-64
                             c1-490a-944e-6a2adee1a451","uuid":"940feda7-65ee-438b-bd06-d33ba3d734c4","ip":"10.1.1.126","netmask":"255.255.255.0","gateway":"10.1.1.1","mac":"02:00:62:78:00:06","dns1":"20
                             8.74.240.5","dns2":"208.74.247.245","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://1679","isolationUri":"vlan://1679","isSecurityGroupEnabled":false,"name":"GUE
                             ST-PUB"}],"vcpuMaxLimit":16},"hostIp":"10.90.2.111","executeInSequence":false,"wait":0}}]
}
    2017-09-25 11:20:16,555 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-378:ctx-d30ac189)
Seq 19-8416664753602063711: Executing request
    2017-09-25 11:20:16,560 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-378:ctx-d30ac189)
1. The VM i-23-223-VM is in Starting state.
    2017-09-25 11:20:16,579 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-378:ctx-d30ac189)
Created VM a6739bcc-968f-abe6-d85c-a016de8b0b9f for i-23-223-VM
    2017-09-25 11:20:16,585 WARN  [c.c.h.x.r.CitrixResourceBase] (DirectAgent-378:ctx-d30ac189)
Catch Exception: class com.xensource.xenapi.Types$UuidInvalid due to The uuid you        
                      supplied was invalid.
    The uuid you supplied was invalid.
    
    So in my mind I should see that uuid in /dev/mapper/ but I do not.
    
    So I looked at Dag's post and wanted to pull down some info from sql.
    
    +-------+-----------------------------------------+----------------+----------------------+----------+------------------+------------+--------------------+--------------+---------+-------+-------+----------------+--------------+--------------------------------------+----------+----------+
    | vmid  | vmname                                  | vminstname     | vmdispname      
    | vmacctid | vmacctname       | vmdomainid | vmdomname          | vmofferingid | vmspeed
| vmmem | volid | volname        | volsize      | volpath                              | voltype
 | volstate |
    +-------+-----------------------------------------+----------------+----------------------+----------+------------------+------------+--------------------+--------------+---------+-------+-------+----------------+--------------+--------------------------------------+----------+----------+
    |   223 | DC2FCE                                  | i-23-223-VM    | DC2FCE          
    |       23 | admin            |         18 | fce.coop           |           29 |    2000
|  4096 |   275 | ROOT-223       | 171798691840 | f5aabea5-2a18-4f24-9f8d-d750a907a699 | ROOT
    | Ready    |
    
    This shows me that the volpath is *699
    
    Now that matches up with the only VHD symbolic link I can find back to /dev/mapper/VG_XenStorage--
    
    VG_XenStorage-469b6dcd-8466-3d03-de0e-cc3983e1b6e2]# ls *699
    VHD-f5aabea5-2a18-4f24-9f8d-d750a907a699
    
    I think the parent volume got deleted and I'm left with only the 699 volume seen above.
    
    Do you have any suggestions on recovering from this?
    
    Jeremy
    
    
    -----Original Message-----
    From: Jeremy Peterson [mailto:jpeterson@acentek.net] 
    Sent: Monday, September 25, 2017 10:15 AM
    To: users@cloudstack.apache.org
    Subject: RE: Corrupted HDD
    
    Ok so I will be going through both of the items listed.
    
    But my problem is our xenserver is with iSCSI SAN's so we do not have any VHD files.
    
    # ls VG_XenStorage--469b6dcd--8466--3d03--de0e--cc3983e1b6e2-VHD--f5aabea5--2a18--4f24--9f8d--d750a907a699
    VG_XenStorage--469b6dcd--8466--3d03--de0e--cc3983e1b6e2-VHD--f5aabea5--2a18--4f24--9f8d--d750a907a699
    
    Now I've tried to run a dd on that file to export and try to recover from that its corrupted.
 
    
    I'm grasping here and will be looking at more information.  I need to recover data from
this VG_XenStorage file.
    
    Thank you in advance.  Just wanted to let you know where I am sitting.
    
    Jeremy
    
    
    -----Original Message-----
    From: Dag Sonstebo [mailto:Dag.Sonstebo@shapeblue.com] 
    Sent: Monday, September 25, 2017 3:16 AM
    To: users@cloudstack.apache.org
    Subject: Re: Corrupted HDD
    
    Hi Jeremy,
    
    In addition to Adrian’s walk-through – take a look at the following blog article from
last year – it should give you some more options to try - http://www.shapeblue.com/recovery-of-vms-to-new-cloudstack-instance/

    
    Regards,
    Dag Sonstebo
    Cloud Architect
    ShapeBlue
    
    On 25/09/2017, 03:49, "Adrian Sender" <asender@testlabs.com.au> wrote:
    
        Hi Jeremy,
        
        Some time ago I worked with Citrix in creating a article to mount and recover
        LVMoHBA.
        
        How to Mount Linux LVM Partition in a XenServer Host
        
        http://support.citrix.com/article/CTX117791
        
        The above CTX article only applies to raw LVM LV. In our case the VDIs are LVs
        with VHD headers in it. We'll have to use blktap to create tapdisks to open
        the VHD LV before we can create device maps for the partitions / LVs within
        the VDI.
        
        Up to activating the VG, the article still applies. After activating the
        VG/LVs, we'll need to manually create tapdisk to open the VHD LV before using
        kpartx to create maps from partition tables.
        
        NOTE: I've included output from my test environment for your reference
        
        1. Create tapdisk
        
        # tap-ctl allocate /dev/xen/blktap-2/tapdev0
        
        2. Spawn a new process
        
        # tap-ctl spawn
        
        tapdisk spawned with pid 30383
        
        # tap-ctl list
        30383 - - - -
        - 0 - - -
        
        4. Attach the tapdisk to the process
        
        tap-ctl attach -p PID -m minor
        
        # tap-ctl attach -p 30383 -m 0
        # tap-ctl list
        30383 0 0 - -
        
        
        5. Open the VHD LV
        
        Note you may need to activate the LVM first with lvchange -ay
        
        # tap-ctl open -p 30383 -m 0 -a
        vhd:/dev/VG_XenStorage-b1415cf5-07e2-14fa-cbf8-6490751da242/VHD-182ce043-bf98-45d9-926b-595bbd82ed8a
        
        6. Verify
        
        # tap-ctl list 
        30383 0 0 vhd
        /dev/VG_XenStorage-b1415cf5-07e2-14fa-cbf8-6490751da242/VHD-182ce043-bf98-45d9-926b-595bbd82ed8a
        
        7. Create device maps from partition tables using kpartx
        
        kpartx -av /dev/xen/blktap-2/tapdev0
        
        # kpartx -av /dev/xen/blktap-2/tapdev0
        add map tapdev0p1 (252:5): 0 1024000 linear /dev/xen/blktap-2/tapdev0 2048
        add map tapdev0p2 (252:6): 0 32872448 linear /dev/xen/blktap-2/tapdev0 1026048
        
        NOTE: in this case, it is a CentOS 7 VDI, contains a /boot partition and VG -
        linux and 2 LVs (root, home).
        
        Device maps were created in
        /dev/mapper/tapdev0pN
        
        8. Mount 1st partition
        
        # mount /dev/mapper/tapdev0p1 /mnt/boot
        
        9. Activate VG
        
        # vgchange -a y
        2 logical volume(s) in volume group "linux" now active
        
        # lvscan
        ACTIVE '/dev/linux/root' [13.67 GB] inherit
        ACTIVE '/dev/linux/home' [2.00 GB] inherit
        
        
        10. Mount root and home LVs
        
        mount -t xfs /dev/linux/root /mnt/root
        
        NOTE: XenServer 6.2 kernel does NOT support XFS
        
        mount -t ext4 /dev/linux/home /mnt/home
        
        11. After backing up data, reverse the procedure to release resources
        
        List partition mappings
        
        # kpartx -lv /dev/xen/blktap-2/tapdev0
        tapdev0p1 : 0 1024000 /dev/xen/blktap-2/tapdev0 2048
        tapdev0p2 : 0 32872448 /dev/xen/blktap-2/tapdev0 1026048
        
        De-active the VG (all LVs)
        
        # vgchange -an linux
        0 logical volume(s) in volume group "linux" now active
        
        Delete partition mappings
        
        # kpartx -dv /dev/xen/blktap-2/tapdev0
        del devmap : tapdev0p2
        del devmap : tapdev0p1
        
        12. Release the blktap resources
        
        # tap-ctl list
        30383 0 0 vhd
        /dev/VG_XenStorage-b1415cf5-07e2-14fa-cbf8-6490751da242/VHD-182ce043-bf98-45d9-926b-595bbd82ed8a
        
        # tap-ctl close -p 30383 -m 0
        [root@xenserver-zilblqhl /]# tap-ctl list
        30383 0 0x2 - -
        
        # tap-ctl destroy -p 30383 -m 0
        [root@xenserver-zilblqhl /]# tap-ctl list
        
        Regards,
        Adrian Sender
        
        
        
        -----
    Dag.Sonstebo@shapeblue.com
    www.shapeblue.com
    53 Chandos Place, Covent Garden, London  WC2N 4HSUK @shapeblue
      
     
    
    
Dag.Sonstebo@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
  
 

----- Original Message -----------
        From: Jeremy Peterson <jpeterson@acentek.net>
        To: "users@cloudstack.apache.org" <users@cloudstack.apache.org>
        Sent: Sun, 24 Sep 2017 22:38:29 +0000
        Subject: Corrupted HDD
        
        > So I need to know if anyone can drop a hint on this.
        > 
        > How can i recover data from cloudstack with xenserver.  One of the 
        > storage LUN's got 100% full.  I was carefully deleting un-associated 
        > disks from the LUN.  A disk that had no VM attached.  Then when one 
        > of the VM's rebooted it came online with no bootable disk found.
        > 
        > Now I'm thinking this over that one of those disks were the a master 
        > UUID and thats why my VM's wont boot.  Can that been fixed?
        > 
        > I look in primary storage and can find the UUID of a disk matches a 
        > volume in /dev/mapper/VG.....
        > 
        > Can I pull that disk down and create a VHD to deploy a template and 
        > deploy a new vm from that vhd file?
        > 
        > Can I dd that disk to export to anything to try and recover data?  
        > Office files or PDF's are critical.
        > 
        > Does anyone know if there are any good consulting companies with 
        > Citrix or if Citrix can help with this?
        > 
        > Jeremy
        ------- End of Original Message -------
        
        
    
    

Mime
View raw message