cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Olivier Mauras <>
Subject Re: Cloustack 4.1.0 + GlusterFS
Date Thu, 12 Sep 2013 05:48:02 GMT

It's always interesting seeing people advising against a solution
without actually having it tested :)

Thanks John for sharing your
I guess that you're using XFS as the underlying FS ? Your
write-behind options, clearly show a performance improvement for write
Have you tried with VM cache enabled? What about read

I also hope that qemu native integration will be soon
supported in Cloudstack, performances are definitely boosted :)

BTW are
you happy with the cloudstack KVM integration? Seems, reading this list,
that there's some "needed" features that doesn't work very well with KVM
- HA, VM snapshots,...


On 2013-09-11 16:38, John
Skinner wrote: 

> I currently have GlusterFS deployed into an 8 node
KVM cluster running on CloudStack 4.1 for primary storage. Gluster is
deployed on 28 1TB drives across 2 separate storage appliances using a
distributed-replicated volume with the replica set to 2. The storage
network is 10Gb copper. 
> These are the options I have configured
for the volume in Gluster, most of them are from a Red Hat document on
configuring Red Hat Enterprise Storage for VM hosting: 
> 32 
> performance.cache-size: 1024MB 
performance.write-behind-window-size: 5MB 
> performance.write-behind:
> network.remote-dio: on 
> cluster.eager-lock: enable 
performance.stat-prefetch: off 
> on 
> on 
> performance.quick-read: on 
> Here are
some of the numbers I was getting when benchmarking the storage from the
KVM node directly (not a VM) 
> The below table is in KB/s. The test
is single stream 1GB file utilizing Direct I/O (no cache). I used iozone
to run the benchmark. 
> Write 4k 45729 
> Read 4k 10189 
> Random
Write 4k 31983 
> Random Read 4k 9859 
> Write 16k 182246 
> Read 16k
> Random Write 16k 113026 
> Random Read 16k 37237 
> Write 64k
> Read 64k 125315 
> Random Write 64k 383848 
> Random Read 64k
> Write 256k 567501 
> Read 256k 218413 
> Random Write 256k
> Random Read 256k 229117 
> In the above results, I have the
volume mounted to each KVM host as a FUSE glusterfs file system. They
are added to CloudStack as a shared mount point. In the future it would
be great to utilize GlusterFS qemu libvirt integration with libgfapi so
I could bypass fuse altogether. However, that would require adding that
code to CloudStack to support that. 
> I maybe have 15 or so VMs
running from the storage now and it is still pretty snappy. Need to do
some more testing though and really get it loaded. 
> ----- Original
Message -----
> From: "Rafael Weingartner"
> To: 
Sent: Wednesday, September 11, 2013 8:48:07 AM 
> Subject: Re: Cloustack
4.1.0 + GlusterFS 
> Right now I can think in three main reasons: 

> The first reason, performance (I do not know Gluster and its
> and if the read and write speed are satisfactory). Please
if you can make a 
> test, post the results. 
> Second consistency, I
do not know Gluster, but swift that is also a 
> Distributed File System
is not consistency and they make it pretty clear on 
> their page
> "Swift is a highly
available, distributed, eventually consistent 
> object/blob store...".

> I would not accept to storage my VMs images on a FS that is
> consistent. 
> Third, network, I haven't used this kind
of FS, but I can image that it 
> uses a lot of bandwidth to keep
synchronizing, managing and securing the 
> data. So, managing the
networking would be a pain. 
> 2013/9/11 Olivier Mauras
>> Hi, Those thinking that it's not a good
idea, do you mind explaining your point of view? GlusterFS seems like
the only real alternative to a highly priced SAN for the primary
storage... Thanks, Olivier On 2013-09-11 15:08, Rafael Weingartner
>>> I
>> totally agree with Tomasz, I do not think that
using a distributed FS as 
>>> primary storage is a good idea, but
as a secondary it sounds
>> interesting. And maybe as a Swift backend.
-- @shankerbalan M: +91 98860 60539 | O: +91 (80) 67935867 [1] [2] | [2][3] |
Twitter:@shapeblue ShapeBlue Services India LLP, 22nd floor, Unit 2201A,
World Trade Centre, Bangalore - 560 055 This email and any attachments
to it may be confidential and are intended solely for the use of the
>>> sarily represent those of Shape Blue Ltd or related
companies. If you are not the intended recipient of this email, you must
>> action based upon its contents, nor copy or show it to anyone.
Please contact the sender if you believe you have received this email in
error. Shape Blue Ltd is a company incorporated in England & Wales.
ShapeBlue Services India LLP is operated under license from Shape Blue
Ltd. ShapeBlue is a registered trademark. -- Rafael Weingartner Links:
------ [1] [3] [2] [4] [3]
> -- Rafael Weingartner



  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message