couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Roger Binns <rog...@rogerbinns.com>
Subject Re: [jira] Commented: (COUCHDB-623) File format for views is space and time inefficient - use a better one
Date Wed, 13 Jan 2010 23:18:18 GMT
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Chris Anderson wrote:
>> It sounds like Couch does just fine if you run compaction. Perhaps we
>> should recommend view compaction more aggressively.

The compacted file sizes are still a little large, but tolerable.  The
problem is that view data is not replicated.  So I can't for example
generate a view, reduce it and then replicate to 50 machines.  If the view
required 20GB to generate then each of those machines needs an additional
20GB of storage, not to mention all the CPU time and I/O.

BTW In my case I am not using reduce and its usage also seems far rarer, so
deliberately using a file format that consumes an excessive amount of space
just in case is not a good optimization.  And for the vast majority of
people the reduce doesn't need transactional levels of consistency anyway.
For example having tag counts be completely accurate (to some point in time)
by consuming 27GB of space for a 1GB data set is not good!

Roger
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAktOVLkACgkQmOOfHg372QQ0UACgqBPDOfGw+ViIPTHINMfGr79C
hJ4AnRQNxUoQ5CnPlxkg+giJa3w8GPnf
=Dscn
-----END PGP SIGNATURE-----


Mime
View raw message