Phil Steitz (JIRA) a écrit :
> I agree this is a corner case and the negative result is due to rounding. The question
is, should we force the result to 0 when a negative value is returned by the computation?
Yes, I think this would be wiser than what I said in my previous
comment. I forgot that people often compute the square root of such
sums, mainly to compare them with some dimensioning threshold.
We really know that the value must be positive and that if it is
negative, this is due to cancellation. Hence we could force the value in
this case.
Just replace the return statement by :
Math.amx(0, sumYY  sumXY * sumXY / sumXX);
... and fix my test case by replacing ther " < 1.0e14" back to ">= 0".
Sorry, I didn't reflect enough
Luc

To unsubscribe, email: commonsdevunsubscribe@jakarta.apache.org
For additional commands, email: commonsdevhelp@jakarta.apache.org
