cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John Skinner <>
Subject Re: Cloustack 4.1.0 + GlusterFS
Date Wed, 11 Sep 2013 14:38:33 GMT
I currently have GlusterFS deployed into an 8 node KVM cluster running on CloudStack 4.1 for
primary storage. Gluster is deployed on 28 1TB drives across 2 separate storage appliances
using a distributed-replicated volume with the replica set to 2. The storage network is 10Gb

These are the options I have configured for the volume in Gluster, most of them are from a
Red Hat document on configuring Red Hat Enterprise Storage for VM hosting: 32 
performance.cache-size: 1024MB 
performance.write-behind-window-size: 5MB 
performance.write-behind: on 
network.remote-dio: on 
cluster.eager-lock: enable 
performance.stat-prefetch: off on on 
performance.quick-read: on 

Here are some of the numbers I was getting when benchmarking the storage from the KVM node
directly (not a VM) 

The below table is in KB/s. The test is single stream 1GB file utilizing Direct I/O (no cache).
I used iozone to run the benchmark. 

Write 4k 	45729 
Read 4k 	10189 
Random Write 4k 	31983 
Random Read 4k 	9859 
Write 16k 	182246 
Read 16k 	37146 
Random Write 16k 	113026 
Random Read 16k 	37237 
Write 64k 	420908 
Read 64k 	125315 
Random Write 64k 	383848 
Random Read 64k 	125218 
Write 256k 	567501 
Read 256k 	218413 
Random Write 256k 	508650 
Random Read 256k 	229117 

In the above results, I have the volume mounted to each KVM host as a FUSE glusterfs file
system. They are added to CloudStack as a shared mount point. In the future it would be great
to utilize GlusterFS qemu libvirt integration with libgfapi so I could bypass fuse altogether.
However, that would require adding that code to CloudStack to support that. 

I maybe have 15 or so VMs running from the storage now and it is still pretty snappy. Need
to do some more testing though and really get it loaded. 

----- Original Message -----

From: "Rafael Weingartner" <> 
Sent: Wednesday, September 11, 2013 8:48:07 AM 
Subject: Re: Cloustack 4.1.0 + GlusterFS 

Right now I can think in three main reasons: 

The first reason, performance (I do not know Gluster and its performance 
and if the read and write speed are satisfactory). Please if you can make a 
test, post the results. 

Second consistency, I do not know Gluster, but swift that is also a 
Distributed File System is not consistency and they make it pretty clear on 
their page ( 

"Swift is a highly available, distributed, eventually consistent 
object/blob store...". 

I would not accept to storage my VMs images on a FS that is eventually 

Third, network, I haven't used this kind of FS, but I can image that it 
uses a lot of bandwidth to keep synchronizing, managing and securing the 
data. So, managing the networking would be a pain. 

2013/9/11 Olivier Mauras <> 

> Hi, 
> Those thinking that it's not a good idea, do you mind 
> explaining your point of view? 
> GlusterFS seems like the only real 
> alternative to a highly priced SAN for the primary storage... 
> Thanks, 
> Olivier 
> On 2013-09-11 15:08, Rafael Weingartner wrote: 
> > I 
> totally agree with Tomasz, I do not think that using a distributed FS 
> as 
> > primary storage is a good idea, but as a secondary it sounds 
> interesting. 
> > 
> > But, off course you can try *;*) 
> > 
> > 2013/9/11 
> Shanker Balan <> 
> > 
> >> On 11-Sep-2013, at 
> 5:14 PM, Tomasz Zięba < [1]> wrote: 
> >> 
> >>> Hi, Some 
> time ago I tried to use GlusterFS as storage for cloudstacka but I 
> noticed that cloudstack uses the default settings for mount command. By 
> default mount command is using the UDP protocol but glusterfs works 
> >> 
> only 
> >> 
> >>> using tcp. I think, if cloudstack developers could add "-o 
> proto=tcp" to code glusterfs should works. For example: /bin/mount -t 
> nfs -o proto=tcp IP:/share /mnt/gluster/ If you are using CitrixXen you 
> should mount the share and make it as SR. For cloudstacka is clear 
> because you should use the option PreSetup when creating PrimaryStorage. 
> Personally, I doubt that using GlusterFS as a primary storage is a good 
> solution but for secondary storage it should be very usefull. 
> >> And 
> maybe as a Swift backend. -- @shankerbalan M: +91 98860 60539 | O: +91 
> (80) 67935867 [2] | [3] | 
> Twitter:@shapeblue ShapeBlue Services India LLP, 22nd floor, Unit 2201A, 
> World Trade Centre, Bangalore - 560 055 This email and any attachments 
> to it may be confidential and are intended solely for the use of the 
> individual to whom it is addressed. Any views or opinions expressed are 
> solely those of the author and do not necessarily represent those of 
> Shape Blue Ltd or related companies. If you are not the intended 
> recipient of this email, you must neither take any action based upon its 
> contents, nor copy or show it to anyone. Please contact the sender if 
> you believe you have received this email in error. Shape Blue Ltd is a 
> company incorporated in England & Wales. ShapeBlue Services India LLP is 
> operated under license from Shape Blue Ltd. ShapeBlue is a registered 
> trademark. 
> > 
> > -- Rafael Weingartner 
> Links: 
> ------ 
> [1] 
> [2] 
> [3] 

Rafael Weingartner 

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message