groovy-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Bond <sbyrne...@gmail.com>
Subject Re: float/double calculation bug ?
Date Sun, 19 Mar 2017 15:47:02 GMT
What happens if you use .toDouble() instead of .toFloat, and an explicitly
double 100.0 literal?  Right now, your computation is taking an integer,
dividing by a float, then multiplying by another float.  I suspect that
going double precision won't fix this in all cases (floating point math is,
after all, just an approximation), but at least you're processing the
values as precisely as you can.

On Sun, Mar 19, 2017 at 8:28 AM, Derek Visch <derek.visch@gmail.com> wrote:

> Looks like floating point to me, what are you expecting?
>
> On Mar 19, 2017 10:04 AM, "Tx. T" <txt8888@yahoo.com> wrote:
>
>> Any idea why the follow code "calc" returns the "70%" of the 330000 incorrectly?
>>
>> testing on: Groovy Version: 2.4.9 JVM: 1.8.0_112 Vendor: Oracle Corporation OS: Mac
OS X
>>
>> Mac OS X 10.12.3
>>
>>
>>
>> groovy:000>     def calc = { amount, ttl ->
>> groovy:001>         double rtn
>> groovy:002>         if (amount[-1] != '%') rtn = amount.toDouble()
>> groovy:003>         else rtn = ttl / 100.0 * amount.replaceAll(/%\Z/, '').toFloat()
>> groovy:004>
>> groovy:004>         rtn
>> groovy:005>     }
>> ===> groovysh_evaluate$_run_closure1@39fcbef6*groovy:000> calc("70%", 330000)
>> ===> 230999.99999999997*
>> groovy:000> calc("10%", 330000)
>> ===> 33000.0
>> groovy:000> calc("20%", 330000)
>> ===> 66000.0
>> groovy:000> calc("30%", 330000)
>> ===> 99000.0
>> groovy:000> calc("40%", 330000)
>> ===> 132000.0
>> groovy:000> calc("50%", 330000)
>> ===> 165000.0
>> groovy:000> calc("60%", 330000)
>> ===> 198000.0*groovy:000> calc("70%", 330000)
>> ===> 230999.99999999997*
>> groovy:000> calc("80%", 330000)
>> ===> 264000.0
>> groovy:000> calc("90%", 330000)
>> ===> 297000.0
>> groovy:000> calc("100%", 330000)
>> ===> 330000.0
>>
>>

Mime
View raw message