cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John Skinner <john.skin...@appcore.com>
Subject Re: Cloustack 4.1.0 + GlusterFS
Date Thu, 12 Sep 2013 00:27:39 GMT
I ran each test independently for each block size. 

iozone -I -i 0 -i 1 -i 2 -r 4k -s 1G 

In order: -I to specify direct-IO, -i 0 to specify write/rewrite, -i 1 to specify read/re-read,
-i 2 to specify random read/write, -r specify block size, -s to specify file size. 

The report is pretty thorough. What I put in the email was different, I just took the throughput
from each test and put it into a table outside of the full report. 

----- Original Message -----

From: "Rafael Weingartner" <rafaelweingartner@gmail.com> 
To: users@cloudstack.apache.org 
Sent: Wednesday, September 11, 2013 4:16:23 PM 
Subject: Re: Cloustack 4.1.0 + GlusterFS 

I have never used iozone before, 

How did you get that report? 

I tried: iozone -s 1024m -t 1 -R 

But the report was pretty different from yours. 


2013/9/11 John Skinner <john.skinner@appcore.com> 

> I currently have GlusterFS deployed into an 8 node KVM cluster running on 
> CloudStack 4.1 for primary storage. Gluster is deployed on 28 1TB drives 
> across 2 separate storage appliances using a distributed-replicated volume 
> with the replica set to 2. The storage network is 10Gb copper. 
> 
> These are the options I have configured for the volume in Gluster, most of 
> them are from a Red Hat document on configuring Red Hat Enterprise Storage 
> for VM hosting: 
> 
> 
> 
> performance.io-thread-count: 32 
> performance.cache-size: 1024MB 
> performance.write-behind-window-size: 5MB 
> performance.write-behind: on 
> network.remote-dio: on 
> cluster.eager-lock: enable 
> performance.stat-prefetch: off 
> performance.io-cache: on 
> performance.read-ahead: on 
> performance.quick-read: on 
> 
> Here are some of the numbers I was getting when benchmarking the storage 
> from the KVM node directly (not a VM) 
> 
> The below table is in KB/s. The test is single stream 1GB file utilizing 
> Direct I/O (no cache). I used iozone to run the benchmark. 
> 
> Write 4k 45729 
> Read 4k 10189 
> Random Write 4k 31983 
> Random Read 4k 9859 
> Write 16k 182246 
> Read 16k 37146 
> Random Write 16k 113026 
> Random Read 16k 37237 
> Write 64k 420908 
> Read 64k 125315 
> Random Write 64k 383848 
> Random Read 64k 125218 
> Write 256k 567501 
> Read 256k 218413 
> Random Write 256k 508650 
> Random Read 256k 229117 
> 
> In the above results, I have the volume mounted to each KVM host as a FUSE 
> glusterfs file system. They are added to CloudStack as a shared mount 
> point. In the future it would be great to utilize GlusterFS qemu libvirt 
> integration with libgfapi so I could bypass fuse altogether. However, that 
> would require adding that code to CloudStack to support that. 
> 
> I maybe have 15 or so VMs running from the storage now and it is still 
> pretty snappy. Need to do some more testing though and really get it loaded. 
> 
> ----- Original Message -----

> 
> From: "Rafael Weingartner" <rafaelweingartner@gmail.com> 
> To: users@cloudstack.apache.org 
> Sent: Wednesday, September 11, 2013 8:48:07 AM 
> Subject: Re: Cloustack 4.1.0 + GlusterFS 
> 
> Right now I can think in three main reasons: 
> 
> The first reason, performance (I do not know Gluster and its performance 
> and if the read and write speed are satisfactory). Please if you can make a 
> test, post the results. 
> 
> Second consistency, I do not know Gluster, but swift that is also a 
> Distributed File System is not consistency and they make it pretty clear on 
> their page (http://docs.openstack.org/developer/swift/) 
> 
> "Swift is a highly available, distributed, eventually consistent 
> object/blob store...". 
> 
> I would not accept to storage my VMs images on a FS that is eventually 
> consistent. 
> 
> Third, network, I haven't used this kind of FS, but I can image that it 
> uses a lot of bandwidth to keep synchronizing, managing and securing the 
> data. So, managing the networking would be a pain. 
> 
> 
> 
> 2013/9/11 Olivier Mauras <olivier@core-hosting.net> 
> 
> > 
> > 
> > Hi, 
> > 
> > Those thinking that it's not a good idea, do you mind 
> > explaining your point of view? 
> > GlusterFS seems like the only real 
> > alternative to a highly priced SAN for the primary storage... 
> > 
> > 
> > Thanks, 
> > Olivier 
> > 
> > On 2013-09-11 15:08, Rafael Weingartner wrote: 
> > 
> > > I 
> > totally agree with Tomasz, I do not think that using a distributed FS 
> > as 
> > > primary storage is a good idea, but as a secondary it sounds 
> > interesting. 
> > > 
> > > But, off course you can try *;*) 
> > > 
> > > 2013/9/11 
> > Shanker Balan <shanker.balan@shapeblue.com> 
> > > 
> > >> On 11-Sep-2013, at 
> > 5:14 PM, Tomasz Zięba <t.a.zieba@gmail.com [1]> wrote: 
> > >> 
> > >>> Hi, Some 
> > time ago I tried to use GlusterFS as storage for cloudstacka but I 
> > noticed that cloudstack uses the default settings for mount command. By 
> > default mount command is using the UDP protocol but glusterfs works 
> > >> 
> > only 
> > >> 
> > >>> using tcp. I think, if cloudstack developers could add "-o 
> > proto=tcp" to code glusterfs should works. For example: /bin/mount -t 
> > nfs -o proto=tcp IP:/share /mnt/gluster/ If you are using CitrixXen you 
> > should mount the share and make it as SR. For cloudstacka is clear 
> > because you should use the option PreSetup when creating PrimaryStorage. 
> > Personally, I doubt that using GlusterFS as a primary storage is a good 
> > solution but for secondary storage it should be very usefull. 
> > >> And 
> > maybe as a Swift backend. -- @shankerbalan M: +91 98860 60539 | O: +91 
> > (80) 67935867 shanker.balan@shapeblue.com [2] | www.shapeblue.com [3] | 
> > Twitter:@shapeblue ShapeBlue Services India LLP, 22nd floor, Unit 2201A, 
> > World Trade Centre, Bangalore - 560 055 This email and any attachments 
> > to it may be confidential and are intended solely for the use of the 
> > individual to whom it is addressed. Any views or opinions expressed are 
> > solely those of the author and do not necessarily represent those of 
> > Shape Blue Ltd or related companies. If you are not the intended 
> > recipient of this email, you must neither take any action based upon its 
> > contents, nor copy or show it to anyone. Please contact the sender if 
> > you believe you have received this email in error. Shape Blue Ltd is a 
> > company incorporated in England & Wales. ShapeBlue Services India LLP is 
> > operated under license from Shape Blue Ltd. ShapeBlue is a registered 
> > trademark. 
> > > 
> > > -- Rafael Weingartner 
> > 
> > 
> > 
> > Links: 
> > ------ 
> > [1] 
> > mailto:t.a.zieba@gmail.com 
> > [2] mailto:shanker.balan@shapeblue.com 
> > [3] 
> > http://www.shapeblue.com 
> > 
> 
> 
> 
> -- 
> Rafael Weingartner 
> 
> 


-- 
Rafael Weingartner 


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message