Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@couchdb.apache.org
Received-SPF: pass (nike.apache.org: domain of paul.joseph.davis@gmail.com
 designates 209.85.210.176 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        b=WcWUaFv1V7QSxzC9BA3B0A88useEnYI9J3QhzK/ESHwYjQ/FTBuKtaQdnX6dcuhMzY
         ay/zag6YFBCdWGHhzaKt8Ae8FGSjsAPL5prQ50RDWBLV+NYwKIvDjyyfsHiiMblvfYo7
         1kIC95+Kc+8Mj7Ym6euYVFqtBUEZwfzqyuURY=
MIME-Version: 1.0
In-Reply-To: <4AF64514.8090401@rogerbinns.com>
References: <4AF5FB9E.70300@rogerbinns.com>
 <e282921e0911071523n5633d6b8r6d1dbf5a2b43f684@mail.gmail.com>
	<4AF622D5.9070708@rogerbinns.com>
 <e2111bbb0911071942o54914797h20e1471703e4fde@mail.gmail.com>
	<4AF64514.8090401@rogerbinns.com>
From: Paul Davis <paul.joseph.davis@gmail.com>
Date: Sat, 7 Nov 2009 23:40:45 -0500
Message-ID: <e2111bbb0911072040r31047b33l3525ec11c692b1c9@mail.gmail.com>
Subject: Re: Silent corruption of large numbers
To: user@couchdb.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Roger,

> If there are limits then I'd expect them to be enforced in some way,
> typically some sort of exception. =A0The last thing I would expect is for=
 the
> numbers to be silently corrupted. =A0If strings over 1,024 bytes were ran=
domly
> mutated would that be acceptable?

Its funny you should mention strings. Because if you dig into the
unicode awesomeness you'll see that its quite unspecified on how
strings can be screwed with when passing through various unicode
implementations. CouchDB actually does try and reject strings (that
are 'valid json') when it knows it can't serialize them.

>> As such, relying on the view engine to error out
>
> In this case the limit is in the Javascript view engine. =A0The CouchDB s=
erver
> doesn't have a problem. =A0If the view engine is Python then it won't hav=
e the
> problem either.

Exactly my point earlier. It depends on the view server. Python might
be AOK, but the Spidermonkey view server isn't. This is one of those
things that no one is going to agree on. I wish everyone could be as
awesome as Python here, but most implementations are just gonna do
weird things here. And when you contemplate you might have Ruby, Lua,
Java, Clojure, Bash, Lisp, D, Brainfuck, C++, JavaScript, and Erlang
(I'm tired, otherwise that list would be longer) clients, it gets even
weirder.

>> While not
>> the best answer the only thing I can suggest would be to do as Adam
>> says and store large values as strings and use a Bignum library in the
>> places you need to manipulate such values.
>
> In this case it is just my test suite that trips over the problem and my
> language (Python) does bignums by default. =A0I'm not too bothered that t=
here
> is a failure but rather by the manner of the failure - silent mutation.

It sucks greatly I agree. But these sorts of things are hard to coral.
To actually *fix* this you'd have to convince the Spidermonkey team to
fix their handling of large numbers. And I don't see them adding a
native bignum support to the spec anytime soon which is why I can only
suggest to program defensively. It surely doesn't taste good, but at
some point we're still bound by the registers we allocate, and even
our scripting languages haven't yet abstracted numbers away from base
2.

>> Even if we told the view server to error on such values, what would
>> that error look like? Would everyone be unable to pass a doc with a
>> big num through the view server (depending on language)? Things get
>> messy quick.
>
> I'm old school. =A0I don't think this kind of mutation is acceptable. =A0=
A user
> is sitting behind layers of user interface, CouchDB libraries, JSON
> libraries, HTTP libraries and several other bits of glue. =A0Silently doi=
ng
> this is bad - one day a user will end up with data corruption with seriou=
s
> repercussions.
>
> Using a string analogy what should a view server do if a string is passed=
 in
> that is larger than it wants to handle? =A0Is silent mutation or corrupti=
on ok?

You're not old school. Your worries are extremely well placed. I
completely agree that in a perfect world its absolutely not
acceptable.

Though I would note that this tracks all the way down to things like
atoi() vs strtol(). Sometimes we just ignore edge cases. And seeing as
this is at the 2^64 bit edge case, I'd still recommend to attack the
problem differently if you care about such scenarios. The alternative
is to force all implementations to care, and that (AFAICT) is just not
in the cards right yet.

And I do care about such things. You can find my recent protestations
on unicode handling in JSON at [1].

HTH,
Paul Davis

[1] https://mail.mozilla.org/pipermail/es5-discuss/2009-October/003383.html