cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stefania (JIRA)" <>
Subject [jira] [Commented] (CASSANDRA-9224) Figure out a better default float precision rule for cqlsh
Date Mon, 18 May 2015 05:52:59 GMT


Stefania commented on CASSANDRA-9224:

Thanks for the comments, I've created a new patch for 2.1, see link attached.

I'm also using python 2.7.6, on Ubuntu trusty. The tests are calibrated for the current default
precision of 5, for which there are no rounding errors on my box. 

For the single precision float (val2) I had to increase the float precision to about 7 or
8 in order to start getting rounding errors and for the double precision (val1) I started
getting rounding errors with a precision of around 15 or 16. I think this is expected.

Are you getting rounding errors only with a precision of 16 or even lower? And do you want
to support a precision higher than 5? If so, I don't think it makes much sense to support
more than 6 for floats, but for doubles we can increase to say about 12 (but we'd have to
enhance the patch further to separate the formatting of floats and doubles).

> Figure out a better default float precision rule for cqlsh
> ----------------------------------------------------------
>                 Key: CASSANDRA-9224
>                 URL:
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Tools
>            Reporter: Tyler Hobbs
>            Assignee: Stefania
>              Labels: cqlsh
>             Fix For: 3.x, 2.1.x, 2.2.x
> We currently use a {{DEFAULT_FLOAT_PRECISION}} of 5 in cqlsh with formatting {{'%.*g'
% (float_precision, val)}}.  In practice, this is way too low.  For example, 12345.5 will
show up as 123456.  Since the float precision is used for cqlsh's COPY TO, it's particularly
important that we maintain as much precision as is practical by default.
> There are some other tricky considerations, though.  If the precision is too high, python
will do something like this:
> {noformat}
> > '%.25g' % (12345.5555555555555555,)
> '12345.55555555555474711582'
> {noformat}
> That's not terrible, but it would be nice to avoid if we can.

This message was sent by Atlassian JIRA

View raw message