incubator-couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mike Leddy <m...@loop.com.br>
Subject Re: Which filesystem is best for deploying couchdb and why?
Date Mon, 20 Sep 2010 21:25:29 GMT
With zfs I get very good compression ratios:

canoa:~# zfs get all zfs/couchdb
NAME         PROPERTY              VALUE                  SOURCE
zfs/couchdb  type                  filesystem             -
zfs/couchdb  creation              Tue Aug  3 16:52 2010  -
zfs/couchdb  used                  53.1G                  -
zfs/couchdb  available             36.0G                  -
zfs/couchdb  referenced            53.1G                  -
zfs/couchdb  compressratio         5.87x                  -
zfs/couchdb  mounted               yes                    -
zfs/couchdb  quota                 none                   default
zfs/couchdb  reservation           none                   default
zfs/couchdb  recordsize            128K                   default
zfs/couchdb  mountpoint            /zfs/couchdb           default
zfs/couchdb  sharenfs              off                    default
zfs/couchdb  checksum              on                     default
zfs/couchdb  compression           gzip                   inherited from zfs
zfs/couchdb  atime                 off                    inherited from zfs
zfs/couchdb  devices               on                     default
......

That's 5.87x using gzip (level 6 which is the default). I wouldn't be able
to use my rather small (128Gb) solid state disks without it :-)

Mike


On Sun, 2010-09-19 at 10:42 +0300, Metin Akat wrote:
> Hmm, then I start thinking that btrfs is really better than ext4,
> especially if I want to do things like copy the files in order to
> deploy (fast) another instance with already build view indexes.
> 
> And would the couchdb files compress well?
> 
> On Sat, Sep 18, 2010 at 12:23 PM, Wout Mertens <wout.mertens@gmail.com> wrote:
> > Everything except the 4KB blocksize I would say. While ZFS and Btrfs are very different
on-disk, they are both Copy-On-Write filesystems with extents, compression and cheap snapshots.
I don't know how the 4KB blocksize settings translates onto Btrfs.
> >
> > They are both really suited for the append-only workload CouchDB presents.
> >
> > Wout.
> >
> > On Sep 18, 2010, at 9:49 , Metin Akat wrote:
> >
> >> What part of this blog post is relevant to btrfs?
> >>
> >> On Fri, Sep 17, 2010 at 10:22 PM, Chris Anderson <jchris@apache.org> wrote:
> >>> On Thu, Sep 16, 2010 at 6:22 PM, Tyler Gillies <tyler@pdxbrain.com>
wrote:
> >>>> Wow, thanks for the thought out writeup!
> >>>>
> >>>
> >>> here's a blog post http://letsgetdugg.com/2010/06/25/couchdb-on-zfs/
> >>>
> >>>> On Thu, Sep 16, 2010 at 12:47 PM, Randall Leeds <randall.leeds@gmail.com>wrote:
> >>>>
> >>>>> Disclaimer: I'm no file systems expert.
> >>>>>
> >>>>> I recommend something with extents otherwise you might take a big
> >>>>> performance hit while couch deletes old db files after compaction.
> >>>>> Compression sounds cool as long as you can do it really fast (are
> >>>>> there setups where this happens in hardware?).
> >>>>>
> >>>>> reiserfs:
> >>>>> According to wikipedia it "still uses the big kernel lock (BKL)
— a
> >>>>> global kernel-wide lock" which makes performance on multiple cores
> >>>>> suffer.
> >>>>> It's big benefit, as I always understood it, is being able to pack
> >>>>> smile files together into single blocks. You will likely not have
lots
> >>>>> of small files with Couch :-P
> >>>>>
> >>>>> xfs:
> >>>>> Delayed allocation might be a big performance win with a Couch.
Since
> >>>>> outstanding writes are committed together in chunks and then fsync'd
> >>>>> all together I bet this feature would do good things for Couch
> >>>>> performance.
> >>>>>
> >>>>> ext(3|4)
> >>>>> I'd recommend ext4 over ext3. Delayed allocation like xfs as well
as
> >>>>> the multiblock allocator should make it much better than ext3. You
> >>>>> also get extents.
> >>>>>
> >>>>> btrfs/zfs:
> >>>>> Some of the features of each sound interesting, but nothing that
> >>>>> stands out to me as "great for CouchDB". Snapshots and backups are
> >>>>> cool, but Couch is doing this for you already in a sense due to
the
> >>>>> way the btree is appended: CouchDB documents are, in a sense,
> >>>>> copy-on-write. Checksumming is cool if you think it's important
for
> >>>>> your data integrity. If you want snapshots for backup you can always
> >>>>> use CouchDB replication.
> >>>>>
> >>>>> If you run any tests I'd be very, very interested in seeing your
results.
> >>>>>
> >>>>> -Randall
> >>>>>
> >>>>> On Thu, Sep 16, 2010 at 03:11, Metin Akat <akat.metin@gmail.com>
wrote:
> >>>>>> I'm sure almost everybody out there is using ext4/3 (including
me),
> >>>>>> but what about filesystems like btrfs, zfs, reiserfs, xfs. Some
of
> >>>>>> them have very appealing feature-sets (like compression for
example,
> >>>>>> and we all know how greedy is couchdb for disk space).
> >>>>>> And I know that for example btrfs is not yet "recommended for
> >>>>>> production". But its time is coming. From what I see, Ubuntu
10.10
> >>>>>> works flawlessly on btrfs.
> >>>>>> So I'd be happy if we have some discussion on the topic, instead
of
> >>>>>> "everybody uses ext4, just use it" kind of stuff :).
> >>>>>> Couchdb was "alpha software" for years, and we all used it in
> >>>>>> production, so we are not afraid of alpha/beta software, as
long as
> >>>>>> it's good :)
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> http://www.readwriteweb.com/about#tyler
> >>>>
> >>>> Ask me anything <http://tumble.pdxbrain.com/ask>!
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> Chris Anderson
> >>> http://jchrisa.net
> >>> http://couch.io
> >>>
> >
> >
> 



Mime
View raw message