cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Praveen Buravilli <Praveen.Buravi...@citrix.com>
Subject RE: Adding ceph RBD storage failed in CentOS 6.5 KVM
Date Fri, 20 Jun 2014 08:32:27 GMT
Hi Andrija,

Thanks for providing detailed instructions. 
I have executed all of the steps given in http://pastebin.com/HwCZEASR and also in http://admintweets.com/centos-kvm-and-ceph-client-side-setup/.
But still facing the same issue. Any other ideas? 

Also, when I tried to install qemu-* rpms, it reported lot of dependency issues(attaching
the output file). 

Please note that "qemu-img" was reporting support for rbd earlier also. Probably ceph osd
node was also running on the same KVM node which might have updated "rbd" in list of supported
formats.

Error:
====
[root@kvm-ovs-002 src-DONT-TOUCH]# virsh pool-define /tmp/rbd.xml
error: Failed to define pool from /tmp/rbd.xml
error: internal error: missing backend for pool type 8 (rbd)

[root@kvm-ovs-002 src-DONT-TOUCH]# qemu-img | grep rbd
Supported formats: raw cow qcow vdi vmdk cloop dmg bochs vpc vvfat qcow2 qed vhdx parallels
nbd blkdebug host_cdrom host_floppy host_device file gluster gluster gluster gluster rbd

Thanks,
Praveen Kumar Buravilli


-----Original Message-----
From: Andrija Panic [mailto:andrija.panic@gmail.com] 
Sent: 20 June 2014 14:35
To: users@cloudstack.apache.org
Subject: Re: Adding ceph RBD storage failed in CentOS 6.5 KVM

Been there, done that....:

This libvirt error "error: internal error missing backend for pool type 8"
means that libvirt was not compiled with RBD backend support.

Here are my steps to compile libvirt 1.2.3 few months ago - change configure options if you
want, I tried to use as much options as possible.
http://pastebin.com/HwCZEASR (note the "ceph-devel" must be installed as in instructions in
order to be able to compile with RBD support)

Also note, that if you are using CentOS 6.5, the "-s" flag was removed from qemu-img packages,
meaning you will not be able to use snapshot functionality on cloudstack (not related to libbvirt),
this is ANY snapshoting will be broken - there is workarround down :)


Also, beside making sure libvirt can talk to RBD/CEPH, you MUST be sure your qemu-img and
qemu-kvm was compiled with RBD support, check like this:
qemu-img | grep "Supported formats"
should get somwething like this - note the "rbd" on the end of output:

*Supported formats: raw cow qcow vdi vmdk cloop dmg bochs vpc vvfat qcow2 qed parallels nbd
blkdebug host_cdrom host_floppy host_device file rbd*


If both qemu and libvirt are fine, you will be able to add CEPH to ACS 4.2 or newer.

If you have stock (CentOS) versions of qemu-img and qemu-kvm - they are NOT compiled with
RBD support, so you will not be able to use CEPH.
You need to install Intank's version of RPM packages, that are based on official RedHat stock
code of those packages, but are pathced for RBD/CEPH support.

Refer to http://admintweets.com/centos-kvm-and-ceph-client-side-setup/ in order to download
Intanks's RPMs - note that the latest RPMs you will find are probably also based on RHEL 6.5
version, that is missing "-s" flag, so you will still NOT be able to use disk snapshotting
in ACS...

I solved this by installing a little bit older RPMs from Intanks (qemu-img and qemu-kvm that
are based on RHEL 6.2 that still has that famous "-s"
flag preseent), please let me know if you need this provided, since they are NOT present on
the Intanks download page at the moment exact versions of packages I installed and which are
working fine for me:
qemu-kvm-tools-0.12.1.2-2.415.el6.3ceph.x86_64
qemu-img-0.12.1.2-2.355.el6.2.cuttlefish.x86_64
qemu-kvm-0.12.1.2-2.415.el6.3ceph.x86_64

This whole setup is working fine for me...

Hope that helps, was going through the same pain, as you do now... :)

Best,
Andrija Panic




On 20 June 2014 04:38, Praveen Buravilli <Praveen.Buravilli@citrix.com>
wrote:

>  Hi,
>
>
>
> I am facing an issue in adding ceph RBD storage to CloudStack. It is 
> failing with “Failed to add datasource” error.
>
> I have followed all the available instructions related to CloudStack, 
> KVM and ceph storage integration.
>
>
>
> CenOS 6.5 KVM is used as KVM node here. I have read in some blog that 
> we need to compile LibVirt in CenOS KVM nodes to make Ceph storage to 
> work with in CloudStack.
>
> Hence I have got git cloned LibVirt package from its source and 
> upgraded LibVirt and Qemu versions.
>
> (Commands used à git cone #####, ./autogen.sh, make, make install).
>
>
>
> It seems CentOS 6.5 KVM needs to be enabled for RBD (driver) support 
> which needs to be specified as a parameter while compiling LibVirt.
>
>
>
> Can anyone throw some pointers on how to rectify this problem?
>
>
>
> *Management Server Exception:*
>
> 2014-06-20 09:58:03,757 DEBUG [agent.transport.Request]
> (catalina-exec-6:null) Seq 1-1602164611: Received:  { Ans: , MgmtId:
> 52234925782, via: 1, Ver: v1, Flags: 10, { Answer } }
>
> 2014-06-20 09:58:03,757 DEBUG [agent.manager.AgentManagerImpl]
> (catalina-exec-6:null) Details from executing class
> com.cloud.agent.api.ModifyStoragePoolCommand: 
> java.lang.NullPointerException
>
>                 at
> com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.createStoragePo
> ol(LibvirtStorageAdaptor.java:531)
>
>                 at
> com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePo
> ol(KVMStoragePoolManager.java:185)
>
>                 at
> com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePo
> ol(KVMStoragePoolManager.java:177)
>
>
>
> I even tried defining a pool using virsh command even that is failing 
> with “internal error missing backend for pool type 8”.
>
> This indicates my KVM LibVirt is not supported for RBD.
>
>
>
> *Virsh exception on manual pool definition*
>
> <pool type='rbd'>
>
> <name>c574980a-19fc-37e9-b6e3-788a7439575d</name>
>
> <uuid>c574980a-19fc-37e9-b6e3-788a7439575d</uuid>
>
> <source>
>
> <host name='192.168.153.25' port='6789'/>
>
> <name>cloudstack</name>
>
> <auth username='cloudstack' type='ceph'>
>
> <secret uuid='c574980a-19fc-37e9-b6e3-788a7439575d'/>
>
> </auth>
>
> </source>
>
> </pool>
>
>
>
> [root@kvm-ovs-002 agent]# virsh pool-define /tmp/rbd.xml
>
> error: Failed to define pool from /tmp/rbd.xml
>
> error: internal error missing backend for pool type 8
>
>
>
> The Ceph storage is working fine and confirmed with following 
> statistics of it.
>
>
>
> *Ceph output*
>
> [root@kvm-ovs-002 ~]# ceph auth list
>
> installed auth entries:
>
>
>
> osd.0
>
>         key: AQCwTKFTSOudGhAAsWAMRFuCqHjvTQKEV0zjvw==
>
>         caps: [mon] allow profile osd
>
>         caps: [osd] allow *
>
> client.admin
>
>         key: AQBRQqFTWOjBKhAA2s7KnL1z3h7PuKeqXMd7SA==
>
>         caps: [mds] allow
>
>         caps: [mon] allow *
>
>         caps: [osd] allow *
>
> client.bootstrap-mds
>
>         key: AQBSQqFTYKm6CRAAjjZotpN68yJaOjS2QTKzKg==
>
>         caps: [mon] allow profile bootstrap-mds
>
> client.bootstrap-osd
>
>         key: AQBRQqFT6GzXNxAA4ZTmVX6LIu0k4Sk7bh2Ifg==
>
>         caps: [mon] allow profile bootstrap-osd
>
> client.cloudstack
>
>         key: AQBNTaFTeCuwFRAA0NE7CCm9rwuq3ngLcGEysQ==
>
>         caps: [mon] allow r
>
>         caps: [osd] allow rwx pool=cloudstack
>
>
>
> [root@ceph ~]# ceph status
>
>     cluster 9c1be0b6-f600-45d7-ae0f-df7bcd3a82cd
>
>      health HEALTH_WARN 292 pgs degraded; 292 pgs stale; 292 pgs stuck 
> stale; 292 pgs stuck unclean; 1/1 in osds are down; clock skew 
> detected on
> mon.kvm-ovs-00
>                               2
>
>      monmap e1: 2 mons at {ceph=
> 192.168.153.25:6789/0,kvm-ovs-002=192.168.160.3: 6789/0}, election 
> epoch 10, quorum 0,1 ceph,kvm-ovs-002
>
>      osdmap e8: 1 osds: 0 up, 1 in
>
>       pgmap v577: 292 pgs, 4 pools, 0 bytes data, 0 objects
>
>             26036 MB used, 824 GB / 895 GB avail
>
>                  292 stale+active+degraded
>
>
>
> [root@kvm-ovs-002 agent]# cat /etc/redhat-release
>
> CentOS release 6.5 (Final)
>
>
>
> The compiled LibVirt is showing upgraded version in virsh but still 
> finding old rpm packages in KVM.
>
> Give me some hint on whether to clean-up these old RPMs?
>
>
>
> *Virsh version*
>
> [root@kvm-ovs-002 agent]# virsh version
>
> Compiled against library: libvirt 1.2.6
>
> Using library: libvirt 1.2.6
>
> Using API: QEMU 1.2.6
>
> Running hypervisor: QEMU 0.12.1
>
>
>
> [root@kvm-ovs-002 agent]# rpm -qa | grep qemu
>
> qemu-kvm-tools-0.12.1.2-2.415.el6_5.10.x86_64
>
> qemu-kvm-0.12.1.2-2.415.el6.3ceph.x86_64
>
> gpxe-roms-qemu-0.9.7-6.10.el6.noarch
>
> qemu-kvm-0.12.1.2-2.415.el6_5.10.x86_64
>
> qemu-img-0.12.1.2-2.415.el6.3ceph.x86_64
>
> qemu-img-0.12.1.2-2.415.el6_5.10.x86_64
>
> qemu-kvm-tools-0.12.1.2-2.415.el6.3ceph.x86_64
>
> qemu-guest-agent-0.12.1.2-2.415.el6_5.10.x86_64
>
> You have mail in /var/spool/mail/root
>
> [root@kvm-ovs-002 agent]# rpm -qa | grep libvirt
>
> libvirt-python-0.10.2-29.el6_5.9.x86_64
>
> libvirt-java-0.4.9-1.el6.noarch
>
> libvirt-cim-0.6.1-9.el6_5.1.x86_64
>
> libvirt-client-0.10.2-29.el6_5.9.x86_64
>
> libvirt-devel-0.10.2-29.el6_5.9.x86_64
>
> fence-virtd-libvirt-0.2.3-15.el6.x86_64
>
> libvirt-0.10.2-29.el6_5.9.x86_64
>
> libvirt-snmp-0.0.2-4.el6.x86_64
>
>
>
> *Attached all the log files from management and kvm servers.
>
>
>
> Thanks,
>
> *Praveen Kumar Buravilli*
>
>
>



-- 

Andrija Panić
--------------------------------------
  http://admintweets.com
--------------------------------------
Mime
View raw message