incubator-couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Newson <robert.new...@gmail.com>
Subject Re: Fixed precision of floating point number not respected in views
Date Tue, 19 Feb 2013 21:23:23 GMT
Yes, it's expected. Your claims to see clean "17.3" are all based on
functions that round the internal representation for printing. Showing
the full precision shows that 17.3 is held only approximately (since
it cannot be represented exactly using a 64-bit floating point
number).

The numbers are not being changed, you are simply being exposed to the truth. :)

This site can show you clearer than I can:
http://www.h-schmidt.net/FloatConverter/IEEE754.html

try seeing the mantissa for 1.0, 1.5, and 1.3, in that order. You'll
notice the first two encode precisely, and the third does not. (1.3
being 1.2999999523162842 really)

B.

On 19 February 2013 21:15, Luca Morandini <lmorandini@ieee.org> wrote:
> On 02/20/2013 08:10 AM, Robert Newson wrote:
>>
>> (17.0 + 0.3) .toPrecision(21)
>
>
> This is to be expected, sin't it ?
>> (17.0 + 0.3) .toPrecision(17)
> '17.300000000000001'
>> (17.0 + 0.3) .toPrecision(16)
> '17.30000000000000'
>
> If you push it to the limit, numerical imprecisions are in order.
>
> What troubles me -and will have a hard time explaining to users- is that
> numbers are changed by the mere act of storing them in CouchDB.
>
>
> Regards,
>
> Luca Morandini
> Data Architect - AURIN project
> Department of Computing and Information Systems
> University of Melbourne
> Tel. +61 03 903 58 380
> Skype: lmorandini
>

Mime
View raw message