cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Caleb Call <calebc...@me.com>
Subject Re: Primary Storage
Date Tue, 23 Oct 2012 04:55:15 GMT
If I'm using fiber (which is what we do) I'm going directly to the node not through another
device that then shares it out over much slower iSCSI or NFS.  I realize using fiber is more
expensive for most, but for us it's the cheaper option because our fiber infrastructure is
already built out, our iSCSI not so much.


On Oct 22, 2012, at 10:04 PM, bruce.m@v365.com.au wrote:

> 
> 
> Id suggest every one have a look at www.osnexus.com
> supports fiber,
> 10gb, infiniband using the SCST iSCSI code from
> http://scst.sourceforge.net/
> has NFS and all the good stuff including a
> pretty good GUI, replication (lumbering is not there yet) runs on
> Ubuntu
> 
> On 23.10.2012 11:40, Andreas Huser wrote:
>> Hi,
>> 
>> for
> Cloudstack i use Solaris 11 ZFS + GlusterFS over Infiniband
>> (RDMA).
> That gives the best performance and most scalable Storage.
>> I have
> tasted some different solutions for primary Storage but the
>> most are
> to expensive and for a CloudStack Cluster not economic or
>> have a poor
> performance.
>> 
>> My Configuration:
>> Storage Node: 
>> Supermicro Server
> (Intel Hardware) with Solaris 11 with SSD write and
>> read cache (read
> crucial-m4, write ZeusIOPS) GlusterFS and dualport
>> ConnectX 40Gbit/s
> Infiniband adapter.
>> 
>> I have installed GlusterFS direct on Solaris
> with a modified code.
>> Want you build bigger systems for more then 50
> VMs it is better you
>> split the Solaris and GlusterFS with a separte
> headnode for GlusterFS
>> 
>> That looks like:
>> Solaris ZFS
> Backendstorage with a dataset Volume (Thin Provision) -->
>> ( SRP Target
> attached direct without Infiniband switch to GF Node) 
>> --> GlusterFS
> Node the srp target formatted with xfs filesystem,
>> create a GlusterFS
> Volume --> ( Infiniband over a Mellanox Port
>> Switch) --> Cloudstack
> Node mount glusterFS Volume over RDMA
>> 
>> For the Dataset Volume at the
> ZFS Storage, disable atime and enable
>> compression.
>> (Space reclaim)
> With compression you can shrink the ZFS Volume with
>> command at Linux
> dd /dev/zero or In a Windows VM with sdelete
>> That gives you space left
> on the Primary Storage for deleted Files in
>> a VM or for deleted vhd's
> or vm's in the cloudstack
>> 
>> greeting Andreas
>> 
>> 
>> 
>> 
>> Mit
> freundlichen Grüßen
>> 
>> Andreas Huser
>> Geschäftsführer
>> System
> Engineer / Consultant
>> (Cisco CSE, SMBAM, LCSE, ASAM)
>> 
> ---------------------------------------
>> Zellerstraße 28 - 77654
> Offenburg
>> Tel: +49(781) 12786898
>> Mobil: +49(176) 10308549
>> 
> ahuser@7five-edv.de
>> 
>> 
>> 
>> 
>> ----- Ursprüngliche Mail ----- 
>> 
>> 
> Von: "Outback Dingo" 
>> An: cloudstack-users@incubator.apache.org 
>> 
> Gesendet: Dienstag, 23. Oktober 2012 02:15:16 
>> Betreff: Re: Primary
> Storage 
>> 
>> On Mon, Oct 22, 2012 at 8:09 PM, Ivan Rodriguez wrote: 
>>> 
> Solaris 11 ZFS and yes we tried different setups, raids levels number
> of
>>> SSD cache, ARC zfs options etc etc etc.
>>> 
>>> Cheers
>>> 
>> 
>> 
> VMWare ?? 
>> 
>>> On Tue, Oct 23, 2012 at 11:05 AM, Outback Dingo
> wrote:
>>> 
>>>> On Mon, Oct 22, 2012 at 8:03 PM, Ivan Rodriguez wrote:
>>>> 
>> We are using ZFS, with jbod, not in production yet exporting NFS
> to
>>>>> cloudstack, I'm not really happy about the performance
>>>>> 
> but I think is related to the hardware itself rather than technology,
> we
>>>>> are using intel SR2625UR and Intel 320 SSD, we were evaluating
> gluster as
>>>>> well, but we decided to move away from that path since
> gluster nfs is
>>>> still
>>>>> performing poorly, plus we would like to
> see cloudstack integrating the
>>>>> gluster-fuse module, we haven't
> decided the final storage setup but at
>>>> the
>>>>> moment we had
> better results with ZFS.
>>>>> 
>>>>> 
>>>> 
>>>> question is whos ZFS and
> have you "tweaked" the zfs / nfs config for
>>>> performance
>>>> 
>>>> 
>> 
>>>>> On Tue, Oct 23, 2012 at 10:44 AM, Nik Martin >> >wrote:
>>>> 
>> 
>>>>>> On 10/22/2012 05:49 PM, Trevor Francis wrote:
>>>>>> 
>>>>>>> 
> ZFS looks really interesting to me and I am leaning that way. I am
>>>> 
>>>> considering using FreeNAS, as people seem to be having good luck
> with
>>>>>>> it. Can anyone weigh in here?
>>>>>>> 
>>>>>>> 
>>>>>> My
> personal opinion, I think FreeNAS and OpenFiler have horrible,
>>>> 
> horrible
>>>>>> User Interfaces - not very intuitive, and they both seem
> to be file
>>>> servers
>>>>>> with things like iSCSI targets tacked on
> as an afterthought.
>>>>>> 
>>>>>> Nik
>>>>>> 
>>>>>> 
>>>>>>> Trevor
> Francis
>>>>>>> Partner
>>>>>>> 46 Labs | The PeerEdge Cloud
>>>>>>> 
> http://www.46labs.com |
>>>> http://www.peeredge.net
>>>>>>> 
>>>>>>> 
> 405-362-0046 - Voice | 405-410-4980 - Cell
>>>>>>> trevorgfrancis -
> Skype
>>>>>>> trevor@46labs.com 
>>>>>>> Solutions Provider for the
> Telecom Industry
>>>>>>> 
>>>>>>>> <
>>>>>>> 
> http://www.twitter.**com/peeredge ><
>>>>>>> 
> http://www.**twitter.com/peeredge ><
>>>>>>> 
> http://**www.facebook.com/PeerEdge >
>>>>>>> 
>>>>>>> On Oct 22, 2012, at
> 2:30 PM, Jason Davis wrote:
>>>>>>> 
>>>>>>> ZFS would be an interesting
> setup as you can do the cache pools like
>>>> you
>>>>>>>> would do in
> CacheCade. The problem with ZFS or CacheCade+DRBD is that
>>>>>>>> 
> they
>>>>>>>> really don't scale out well if you are looking for
> something with a
>>>>>>>> unified
>>>>>>>> name space. I'll say however
> that ZFS is a battle hardened FS with
>>>> tons
>>>>>>>> of
>>>>>>>> 
> shops using it. A lot of the whiz-bang SSD+SATA disk SAN things
> these
>>>>>>>> smaller start up companies are hocking are just ZFS
> appliances.
>>>>>>>> 
>>>>>>>> RBD looks interesting but I'm not sure if
> I would be willing to put
>>>>>>>> production data on it, I'm not sure
> how performant it is IRL. From a
>>>>>>>> purely technical perspective,
> it looks REALLY cool.
>>>>>>>> 
>>>>>>>> I suppose anything is fast if
> you put SSDs in it :) GlusterFS is
>>>> another
>>>>>>>> option although
> historically small/random IO has not been it's strong
>>>>>>>> 
> point.
>>>>>>>> 
>>>>>>>> If you are ok spending money on software and
> want a scale out block
>>>>>>>> storage
>>>>>>>> then you might want to
> consider HP LeftHand's VSA product. I am
>>>>>>>> personally
>>>>>>>> 
> partial to NFS plays:) I went the exact opposite approach and
> settled
>>>> on
>>>>>>>> Isilon for our primary storage for our CS
> deployment.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On Mon, Oct 22,
> 2012 at 10:24 AM, Nik Martin >> >>>> >>wrote:
>>>>>>>> 
>>>>>>>> On
> 10/22/2012 10:16 AM, Trevor Francis wrote:
>>>>>>>>> 
>>>>>>>>> We are
> looking at building a Primary Storage solution for an
>>>>>>>>>> 
> enterprise/carrier class application. However, we want to build it
>>>> 
>>>>>>> using
>>>>>>>>>> a FOSS solution and not a commercial solution.
> Do you have a
>>>>>>>>>> recommendation on platform?
>>>>>>>>>> 
>>>> 
>>>>>>> 
>>>>>>>>>> Trevor,
>>>>>>>>> 
>>>>>>>>> I got EXCELLENT results
> builing a SAN from FOSS using:
>>>>>>>>> OS: Centos
>>>>>>>>> Hardware:
> 2X storage servers, with 12x2TB 3.5 SATA drives. LSI
>>>> MegaRAID
>>>> 
>>>>>> with CacheCade Pro, with 240 GB Intel 520 SSDs configured to do
> SSD
>>>>>>>>> caching
>>>>>>>>> (alternately, look at FlashCache from
> Facebook)
>>>>>>>>> intel 10GB dual port nics, one port for crossover,
> on port for up
>>>> link
>>>>>>>>> to
>>>>>>>>> storage network
>>>> 
>>>>>> 
>>>>>>>>> DRBD for real time block replication to
> active-active
>>>>>>>>> Pacemaker+corosync for HA Resource
> management
>>>>>>>>> tgtd for iSCSI target
>>>>>>>>> 
>>>>>>>>> If you
> want file backed storage, XFS is a very good filesystem on
>>>> Linux
>>>> 
>>>>>> now.
>>>>>>>>> 
>>>>>>>>> Pacemaker+Corosync can be difficult to
> grok at the beginning, but
>>>> that
>>>>>>>>> setup gave me a VERY high
> performance SAN. The downside is it is
>>>>>>>>> entirely
>>>>>>>>> 
> managed by CLI, no UI whatsoever.
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Trevor
> Francis
>>>>>>>>>> Partner
>>>>>>>>>> 46 Labs | The PeerEdge Cloud
>>>> 
>>>>>>> http://www.46labs.com |
>>>>>>>>>> http://www.peeredge.net
>>>> 
>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 405-362-0046 - Voice | 405-410-4980 -
> Cell
>>>>>>>>>> trevorgfrancis - Skype
>>>>>>>>>> trevor@46labs.com >>
> trevor@46labs.com
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
> Solutions Provider for the Telecom Industry
>>>>>>>>>> 
>>>>>>>>>>>> 
> http://www.twitter.com/**peeredge><
>>>>>>>>>> 
> http://www.twitter.com/**peeredge >> >>><
>>>>>>>>>> 
> http://www.twitter.**com/**peeredge <
>>>> 
> http://www.twitter.com/**peeredge
>>>>>>>>>>>> <
>>>>>>>>>> 
> http://www.**twitter.com/**peeredge <
>>>>>>>>>> 
> http://www.twitter.com/**peeredge >> >>><
>>>>>>>>>> 
> http://**www.facebook.com/**PeerEdge<
>>>> 
> http://www.facebook.com/PeerEdge><
>>>>>>>>>> 
> http://www.facebook.com/**PeerEdge <
>>>> 
> http://www.facebook.com/PeerEdge>
>>>>>>>>>>>> 
>>>>>>>>>> 
>>>> 
>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>> 


Mime
View raw message