Mailing-List: contact users-help@cloudstack.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@cloudstack.apache.org
Received-SPF: pass (athena.apache.org: local policy)
Date: Wed, 11 Sep 2013 19:27:39 -0500 (CDT)
From: John Skinner <john.skinner@appcore.com>
To: users@cloudstack.apache.org
Message-ID: <474185170.48436.1378945659456.JavaMail.zimbra@appcore.com>
In-Reply-To: 
 <CAG97racYrrAnjwOKjRVKOVE9S4-CwSMmEkG_im0m=qm-gfFuEQ@mail.gmail.com>
References: <1378808766.54031.YahooMailNeo@web163006.mail.bf1.yahoo.com>
 <CAHwdYef=LxeWRu6SdYesOSSCgg9NfEjPFoe0yrsLRgkeqPCwDg@mail.gmail.com>
 <1AA2292B-4B18-4E04-A236-5579EA54D124@shapeblue.com>
 <CAG97racU5k6v7g-_q8+9KodkhpJ-xbo1pyj5AmZ-XLqxW7wJgw@mail.gmail.com>
 <e7d34af8e8dc5e78acb73c82727bd915@core-hosting.net>
 <CAG97rafvmewhu7JtV_8t+HgTvufK+jRfLWfVhTNc1TO3TPrvBw@mail.gmail.com>
 <318558231.34350.1378910313231.JavaMail.zimbra@appcore.com>
 <CAG97racYrrAnjwOKjRVKOVE9S4-CwSMmEkG_im0m=qm-gfFuEQ@mail.gmail.com>
Subject: Re: Cloustack 4.1.0 + GlusterFS
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----=_Part_48435_839370159.1378945659455"
Thread-Topic: Cloustack 4.1.0 + GlusterFS
Thread-Index: R9LccLBA/VDDHz3t0ehlE0Iwd3SgIQ==

------=_Part_48435_839370159.1378945659455
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

I ran each test independently for each block size.=20

iozone -I -i 0 -i 1 -i 2 -r 4k -s 1G=20

In order: -I to specify direct-IO, -i 0 to specify write/rewrite, -i 1 to s=
pecify read/re-read, -i 2 to specify random read/write, -r specify block si=
ze, -s to specify file size.=20

The report is pretty thorough. What I put in the email was different, I jus=
t took the throughput from each test and put it into a table outside of the=
 full report.=20

----- Original Message -----

From: "Rafael Weingartner" <rafaelweingartner@gmail.com>=20
To: users@cloudstack.apache.org=20
Sent: Wednesday, September 11, 2013 4:16:23 PM=20
Subject: Re: Cloustack 4.1.0 + GlusterFS=20

I have never used iozone before,=20

How did you get that report?=20

I tried: iozone -s 1024m -t 1 -R=20

But the report was pretty different from yours.=20


2013/9/11 John Skinner <john.skinner@appcore.com>=20

> I currently have GlusterFS deployed into an 8 node KVM cluster running on=
=20
> CloudStack 4.1 for primary storage. Gluster is deployed on 28 1TB drives=
=20
> across 2 separate storage appliances using a distributed-replicated volum=
e=20
> with the replica set to 2. The storage network is 10Gb copper.=20
>=20
> These are the options I have configured for the volume in Gluster, most o=
f=20
> them are from a Red Hat document on configuring Red Hat Enterprise Storag=
e=20
> for VM hosting:=20
>=20
>=20
>=20
> performance.io-thread-count: 32=20
> performance.cache-size: 1024MB=20
> performance.write-behind-window-size: 5MB=20
> performance.write-behind: on=20
> network.remote-dio: on=20
> cluster.eager-lock: enable=20
> performance.stat-prefetch: off=20
> performance.io-cache: on=20
> performance.read-ahead: on=20
> performance.quick-read: on=20
>=20
> Here are some of the numbers I was getting when benchmarking the storage=
=20
> from the KVM node directly (not a VM)=20
>=20
> The below table is in KB/s. The test is single stream 1GB file utilizing=
=20
> Direct I/O (no cache). I used iozone to run the benchmark.=20
>=20
> Write 4k 45729=20
> Read 4k 10189=20
> Random Write 4k 31983=20
> Random Read 4k 9859=20
> Write 16k 182246=20
> Read 16k 37146=20
> Random Write 16k 113026=20
> Random Read 16k 37237=20
> Write 64k 420908=20
> Read 64k 125315=20
> Random Write 64k 383848=20
> Random Read 64k 125218=20
> Write 256k 567501=20
> Read 256k 218413=20
> Random Write 256k 508650=20
> Random Read 256k 229117=20
>=20
> In the above results, I have the volume mounted to each KVM host as a FUS=
E=20
> glusterfs file system. They are added to CloudStack as a shared mount=20
> point. In the future it would be great to utilize GlusterFS qemu libvirt=
=20
> integration with libgfapi so I could bypass fuse altogether. However, tha=
t=20
> would require adding that code to CloudStack to support that.=20
>=20
> I maybe have 15 or so VMs running from the storage now and it is still=20
> pretty snappy. Need to do some more testing though and really get it load=
ed.=20
>=20
> ----- Original Message -----

>=20
> From: "Rafael Weingartner" <rafaelweingartner@gmail.com>=20
> To: users@cloudstack.apache.org=20
> Sent: Wednesday, September 11, 2013 8:48:07 AM=20
> Subject: Re: Cloustack 4.1.0 + GlusterFS=20
>=20
> Right now I can think in three main reasons:=20
>=20
> The first reason, performance (I do not know Gluster and its performance=
=20
> and if the read and write speed are satisfactory). Please if you can make=
 a=20
> test, post the results.=20
>=20
> Second consistency, I do not know Gluster, but swift that is also a=20
> Distributed File System is not consistency and they make it pretty clear =
on=20
> their page (http://docs.openstack.org/developer/swift/)=20
>=20
> "Swift is a highly available, distributed, eventually consistent=20
> object/blob store...".=20
>=20
> I would not accept to storage my VMs images on a FS that is eventually=20
> consistent.=20
>=20
> Third, network, I haven't used this kind of FS, but I can image that it=
=20
> uses a lot of bandwidth to keep synchronizing, managing and securing the=
=20
> data. So, managing the networking would be a pain.=20
>=20
>=20
>=20
> 2013/9/11 Olivier Mauras <olivier@core-hosting.net>=20
>=20
> >=20
> >=20
> > Hi,=20
> >=20
> > Those thinking that it's not a good idea, do you mind=20
> > explaining your point of view?=20
> > GlusterFS seems like the only real=20
> > alternative to a highly priced SAN for the primary storage...=20
> >=20
> >=20
> > Thanks,=20
> > Olivier=20
> >=20
> > On 2013-09-11 15:08, Rafael Weingartner wrote:=20
> >=20
> > > I=20
> > totally agree with Tomasz, I do not think that using a distributed FS=
=20
> > as=20
> > > primary storage is a good idea, but as a secondary it sounds=20
> > interesting.=20
> > >=20
> > > But, off course you can try *;*)=20
> > >=20
> > > 2013/9/11=20
> > Shanker Balan <shanker.balan@shapeblue.com>=20
> > >=20
> > >> On 11-Sep-2013, at=20
> > 5:14 PM, Tomasz Zi=C4=99ba <t.a.zieba@gmail.com [1]> wrote:=20
> > >>=20
> > >>> Hi, Some=20
> > time ago I tried to use GlusterFS as storage for cloudstacka but I=20
> > noticed that cloudstack uses the default settings for mount command. By=
=20
> > default mount command is using the UDP protocol but glusterfs works=20
> > >>=20
> > only=20
> > >>=20
> > >>> using tcp. I think, if cloudstack developers could add "-o=20
> > proto=3Dtcp" to code glusterfs should works. For example: /bin/mount -t=
=20
> > nfs -o proto=3Dtcp IP:/share /mnt/gluster/ If you are using CitrixXen y=
ou=20
> > should mount the share and make it as SR. For cloudstacka is clear=20
> > because you should use the option PreSetup when creating PrimaryStorage=
.=20
> > Personally, I doubt that using GlusterFS as a primary storage is a good=
=20
> > solution but for secondary storage it should be very usefull.=20
> > >> And=20
> > maybe as a Swift backend. -- @shankerbalan M: +91 98860 60539 | O: +91=
=20
> > (80) 67935867 shanker.balan@shapeblue.com [2] | www.shapeblue.com [3] |=
=20
> > Twitter:@shapeblue ShapeBlue Services India LLP, 22nd floor, Unit 2201A=
,=20
> > World Trade Centre, Bangalore - 560 055 This email and any attachments=
=20
> > to it may be confidential and are intended solely for the use of the=20
> > individual to whom it is addressed. Any views or opinions expressed are=
=20
> > solely those of the author and do not necessarily represent those of=20
> > Shape Blue Ltd or related companies. If you are not the intended=20
> > recipient of this email, you must neither take any action based upon it=
s=20
> > contents, nor copy or show it to anyone. Please contact the sender if=
=20
> > you believe you have received this email in error. Shape Blue Ltd is a=
=20
> > company incorporated in England & Wales. ShapeBlue Services India LLP i=
s=20
> > operated under license from Shape Blue Ltd. ShapeBlue is a registered=
=20
> > trademark.=20
> > >=20
> > > -- Rafael Weingartner=20
> >=20
> >=20
> >=20
> > Links:=20
> > ------=20
> > [1]=20
> > mailto:t.a.zieba@gmail.com=20
> > [2] mailto:shanker.balan@shapeblue.com=20
> > [3]=20
> > http://www.shapeblue.com=20
> >=20
>=20
>=20
>=20
> --=20
> Rafael Weingartner=20
>=20
>=20


--=20
Rafael Weingartner=20


------=_Part_48435_839370159.1378945659455--