couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Roger Binns <rog...@rogerbinns.com>
Subject Re: Silent corruption of large numbers
Date Tue, 10 Nov 2009 07:36:04 GMT
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Paul Davis wrote:
> The only answer that sounds sensible to me is to code defensively and
> make your own guarantees on data passing. The Couch world is only
> going to expand, so data will come and go from a huge multitude of
> sources and through any number of intermediaries. 

The last phrase is what scares me.  My component being defensive won't
protect the ultimate end user from the other parts.

> Or in a more specific case, I don't even know how I would get the JS
> view engine to raise an error on this condition. Except for taking the
> raw string, deserializing, reserializing, and comparing, and even that
> is obviously prone to lots of other oddities of escaping and so on and
> such forth.

I'm advocating work for other people here but perhaps some sort of
test/qualification suite for view servers would do the trick.  It would work
for me having a page listing the CouchDB server itself and then view servers
and what their limits and quirks are as established by the aforementioned
test/qualification suite.

For example the number of significant digits for floating point, integer
ranges, string sizes etc can be listed.  At a guess there are also limits on
string sizes, Unicode capabilities (codepoints above 0xffff), surrogate pair
 handling), JSON items (eg number of items in a JSON list, keys in an object
etc).  I am guessing there are various 32 bit limits around even on 64 bit
platforms.  At some point I'll try storing 5GB strings or lists with more
than 4 billion items and see what happens :-)

> In trunk the computation and index writing are split to take up to two
> cores.

Cool.  Currently what I see is beam.smp and couchjs in total taking up 100%
of a cpu.  Using beam.smp is taking up the most but sometimes it is couchjs
at 53% and beam.smp at 47%.

Roger
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkr5F+IACgkQmOOfHg372QTDpQCePRuT5uP7/jX23U5dh8VcmI/v
AgQAn2foQMt6jHGgU3hvQ7FZt14vfAgb
=Oujm
-----END PGP SIGNATURE-----

Mime
View raw message