cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrija Panic <andrija.pa...@gmail.com>
Subject Re: Adding ceph RBD storage failed in CentOS 6.5 KVM
Date Fri, 20 Jun 2014 06:35:20 GMT
Been there, done that....:

This libvirt error "error: internal error missing backend for pool type 8"
means that libvirt was not compiled with RBD backend support.

Here are my steps to compile libvirt 1.2.3 few months ago - change
configure options if you want, I tried to use as much options as possible.
http://pastebin.com/HwCZEASR (note the "ceph-devel" must be installed as in
instructions in order to be able to compile with RBD support)

Also note, that if you are using CentOS 6.5, the "-s" flag was removed from
qemu-img packages, meaning you will not be able to use snapshot
functionality on cloudstack (not related to libbvirt), this is ANY
snapshoting will be broken - there is workarround down :)


Also, beside making sure libvirt can talk to RBD/CEPH, you MUST be sure
your qemu-img and qemu-kvm was compiled with RBD support, check like this:
qemu-img | grep "Supported formats"
should get somwething like this - note the "rbd" on the end of output:

*Supported formats: raw cow qcow vdi vmdk cloop dmg bochs vpc vvfat qcow2
qed parallels nbd blkdebug host_cdrom host_floppy host_device file rbd*


If both qemu and libvirt are fine, you will be able to add CEPH to ACS 4.2
or newer.

If you have stock (CentOS) versions of qemu-img and qemu-kvm - they are NOT
compiled with RBD support, so you will not be able to use CEPH.
You need to install Intank's version of RPM packages, that are based on
official RedHat stock code of those packages, but are pathced for RBD/CEPH
support.

Refer to http://admintweets.com/centos-kvm-and-ceph-client-side-setup/ in
order to download Intanks's RPMs - note that the latest RPMs you will find
are probably also based on RHEL 6.5 version, that is missing "-s" flag, so
you will still NOT be able to use disk snapshotting in ACS...

I solved this by installing a little bit older RPMs from Intanks (qemu-img
and qemu-kvm that are based on RHEL 6.2 that still has that famous "-s"
flag preseent), please let me know if you need this provided, since they
are NOT present on the Intanks download page at the moment
exact versions of packages I installed and which are working fine for me:
qemu-kvm-tools-0.12.1.2-2.415.el6.3ceph.x86_64
qemu-img-0.12.1.2-2.355.el6.2.cuttlefish.x86_64
qemu-kvm-0.12.1.2-2.415.el6.3ceph.x86_64

This whole setup is working fine for me...

Hope that helps, was going through the same pain, as you do now... :)

Best,
Andrija Panic




On 20 June 2014 04:38, Praveen Buravilli <Praveen.Buravilli@citrix.com>
wrote:

>  Hi,
>
>
>
> I am facing an issue in adding ceph RBD storage to CloudStack. It is
> failing with “Failed to add datasource” error.
>
> I have followed all the available instructions related to CloudStack, KVM
> and ceph storage integration.
>
>
>
> CenOS 6.5 KVM is used as KVM node here. I have read in some blog that we
> need to compile LibVirt in CenOS KVM nodes to make Ceph storage to work
> with in CloudStack.
>
> Hence I have got git cloned LibVirt package from its source and upgraded
> LibVirt and Qemu versions.
>
> (Commands used à git cone #####, ./autogen.sh, make, make install).
>
>
>
> It seems CentOS 6.5 KVM needs to be enabled for RBD (driver) support which
> needs to be specified as a parameter while compiling LibVirt.
>
>
>
> Can anyone throw some pointers on how to rectify this problem?
>
>
>
> *Management Server Exception:*
>
> 2014-06-20 09:58:03,757 DEBUG [agent.transport.Request]
> (catalina-exec-6:null) Seq 1-1602164611: Received:  { Ans: , MgmtId:
> 52234925782, via: 1, Ver: v1, Flags: 10, { Answer } }
>
> 2014-06-20 09:58:03,757 DEBUG [agent.manager.AgentManagerImpl]
> (catalina-exec-6:null) Details from executing class
> com.cloud.agent.api.ModifyStoragePoolCommand: java.lang.NullPointerException
>
>                 at
> com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:531)
>
>                 at
> com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:185)
>
>                 at
> com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:177)
>
>
>
> I even tried defining a pool using virsh command even that is failing with
> “internal error missing backend for pool type 8”.
>
> This indicates my KVM LibVirt is not supported for RBD.
>
>
>
> *Virsh exception on manual pool definition*
>
> <pool type='rbd'>
>
> <name>c574980a-19fc-37e9-b6e3-788a7439575d</name>
>
> <uuid>c574980a-19fc-37e9-b6e3-788a7439575d</uuid>
>
> <source>
>
> <host name='192.168.153.25' port='6789'/>
>
> <name>cloudstack</name>
>
> <auth username='cloudstack' type='ceph'>
>
> <secret uuid='c574980a-19fc-37e9-b6e3-788a7439575d'/>
>
> </auth>
>
> </source>
>
> </pool>
>
>
>
> [root@kvm-ovs-002 agent]# virsh pool-define /tmp/rbd.xml
>
> error: Failed to define pool from /tmp/rbd.xml
>
> error: internal error missing backend for pool type 8
>
>
>
> The Ceph storage is working fine and confirmed with following statistics
> of it.
>
>
>
> *Ceph output*
>
> [root@kvm-ovs-002 ~]# ceph auth list
>
> installed auth entries:
>
>
>
> osd.0
>
>         key: AQCwTKFTSOudGhAAsWAMRFuCqHjvTQKEV0zjvw==
>
>         caps: [mon] allow profile osd
>
>         caps: [osd] allow *
>
> client.admin
>
>         key: AQBRQqFTWOjBKhAA2s7KnL1z3h7PuKeqXMd7SA==
>
>         caps: [mds] allow
>
>         caps: [mon] allow *
>
>         caps: [osd] allow *
>
> client.bootstrap-mds
>
>         key: AQBSQqFTYKm6CRAAjjZotpN68yJaOjS2QTKzKg==
>
>         caps: [mon] allow profile bootstrap-mds
>
> client.bootstrap-osd
>
>         key: AQBRQqFT6GzXNxAA4ZTmVX6LIu0k4Sk7bh2Ifg==
>
>         caps: [mon] allow profile bootstrap-osd
>
> client.cloudstack
>
>         key: AQBNTaFTeCuwFRAA0NE7CCm9rwuq3ngLcGEysQ==
>
>         caps: [mon] allow r
>
>         caps: [osd] allow rwx pool=cloudstack
>
>
>
> [root@ceph ~]# ceph status
>
>     cluster 9c1be0b6-f600-45d7-ae0f-df7bcd3a82cd
>
>      health HEALTH_WARN 292 pgs degraded; 292 pgs stale; 292 pgs stuck
> stale; 292 pgs stuck unclean; 1/1 in osds are down; clock skew detected on
> mon.kvm-ovs-00
>                               2
>
>      monmap e1: 2 mons at {ceph=
> 192.168.153.25:6789/0,kvm-ovs-002=192.168.160.3: 6789/0}, election epoch
> 10, quorum 0,1 ceph,kvm-ovs-002
>
>      osdmap e8: 1 osds: 0 up, 1 in
>
>       pgmap v577: 292 pgs, 4 pools, 0 bytes data, 0 objects
>
>             26036 MB used, 824 GB / 895 GB avail
>
>                  292 stale+active+degraded
>
>
>
> [root@kvm-ovs-002 agent]# cat /etc/redhat-release
>
> CentOS release 6.5 (Final)
>
>
>
> The compiled LibVirt is showing upgraded version in virsh but still
> finding old rpm packages in KVM.
>
> Give me some hint on whether to clean-up these old RPMs?
>
>
>
> *Virsh version*
>
> [root@kvm-ovs-002 agent]# virsh version
>
> Compiled against library: libvirt 1.2.6
>
> Using library: libvirt 1.2.6
>
> Using API: QEMU 1.2.6
>
> Running hypervisor: QEMU 0.12.1
>
>
>
> [root@kvm-ovs-002 agent]# rpm -qa | grep qemu
>
> qemu-kvm-tools-0.12.1.2-2.415.el6_5.10.x86_64
>
> qemu-kvm-0.12.1.2-2.415.el6.3ceph.x86_64
>
> gpxe-roms-qemu-0.9.7-6.10.el6.noarch
>
> qemu-kvm-0.12.1.2-2.415.el6_5.10.x86_64
>
> qemu-img-0.12.1.2-2.415.el6.3ceph.x86_64
>
> qemu-img-0.12.1.2-2.415.el6_5.10.x86_64
>
> qemu-kvm-tools-0.12.1.2-2.415.el6.3ceph.x86_64
>
> qemu-guest-agent-0.12.1.2-2.415.el6_5.10.x86_64
>
> You have mail in /var/spool/mail/root
>
> [root@kvm-ovs-002 agent]# rpm -qa | grep libvirt
>
> libvirt-python-0.10.2-29.el6_5.9.x86_64
>
> libvirt-java-0.4.9-1.el6.noarch
>
> libvirt-cim-0.6.1-9.el6_5.1.x86_64
>
> libvirt-client-0.10.2-29.el6_5.9.x86_64
>
> libvirt-devel-0.10.2-29.el6_5.9.x86_64
>
> fence-virtd-libvirt-0.2.3-15.el6.x86_64
>
> libvirt-0.10.2-29.el6_5.9.x86_64
>
> libvirt-snmp-0.0.2-4.el6.x86_64
>
>
>
> *Attached all the log files from management and kvm servers.
>
>
>
> Thanks,
>
> *Praveen Kumar Buravilli*
>
>
>



-- 

Andrija Panić
--------------------------------------
  http://admintweets.com
--------------------------------------

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message