db-derby-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matthias Ohlemeyer <matthias.ohleme...@web.de>
Subject Re: Bug in division operator with numeric operands?
Date Mon, 20 Mar 2006 23:18:48 GMT
Hi Satheesh,

thanks for all your input! I'm still figuring out the answers to your
first post, but i think I might as well start here:

Although you are right for the current choice of values (operand 1: 1.5,
operand 2: 2.5), the situation does not improve for other operands, e.g.
replace 1.5 with 7 and 2.5 with 17. The SELECT will yield a value of
0.41176... in column1, a value of exactly 0.4 in column 2 (which is
plain wrong!) and about the same value as in column 1 in column 3.

Matthias

Am Montag, den 20.03.2006, 14:23 -0800 schrieb Satheesh Bandaram:
> Also note your script would work if you had used a precision of 30,
> instead of 31.
> 
> ij> CREATE TABLE t (d1 DOUBLE, d2 DOUBLE, n1 NUMERIC(30,11), n2
> NUMERIC(30,11));
> 0 rows inserted/updated/deleted
> ij> INSERT INTO t VALUES (1.5, 2.5, 1.5, 2.5);
> 1 row inserted/updated/deleted
> ij> SELECT d1/d2, n1/n2, n1*(1.0/n2) FROM t;
> 1                     |2                                 |3
> 
> --------------------------------------------------------------------------------
> 0.6                   |0.6                               |
> 0.60000000000000000000
> 
> 1 row selected
> 
> Satheesh
> 
> Satheesh Bandaram wrote:
> > Hi Matthias,
> > 
> > Matthias Ohlemeyer wrote: 
> > > Hi,
> > > 
> > > I've been trying to port a relatively complex application from Oracle to
> > > Derby for quite a while now, but it seems like I've hit a road block
> > > now.
> > > 
> > > Can somebody explain the following behaviour? I copied the following
> > > input-output sequence from ij:
> > >   
> > I can help, if you promise to share our experience of porting your
> > application from Oracle to Derby. :-) 
> > What kind of issues you faced and how easy or difficult was it
> > etc...
> > > CREATE TABLE t (d1 DOUBLE, d2 DOUBLE, n1 NUMERIC(31,11), n2
> > > NUMERIC(31,11));
> > > INSERT INTO t VALUES (1.5, 2.5, 1.5, 2.5);
> > > SELECT d1/d2, n1/n2, n1*(1.0/n2) FROM t;
> > > 
> > > 1           |2                 |3
> > > 
> > > ----------------------------------------------------------------
> > > 0.6         |0                 |0.600000000000000000000000000000
> > > 
> > > 1 row selected
> > >   
> > This seems like a simple bug, at the first look.  If you look at
> > NumericTypeCompiler code, which calculates the scale and precision
> > of operation result type, the comments and the code doesn't seem to
> > match. (getScale() method) I would suggest filing a bug in JIRA.
> > 
> > I can provide more help in resolving the issue.
> > 
> > Satheesh 
> > 
> > NumericTypeCompiler.java
> > 
> >         else if (TypeCompiler.DIVIDE_OP.equals(operator))
> >         {
> >             /*
> >             ** Take max left scale + right precision - right scale +
> > 1, 
> >             ** or 4, whichever is biggest 
> >             */
> >             LanguageConnectionContext lcc =
> > (LanguageConnectionContext)
> > 
> > (ContextService.getContext(LanguageConnectionContext.CONTEXT_ID)); 
> > 
> >             // Scale: 31 - left precision + left scale - right scale
> >             val =
> > Math.max(NumberDataValue.MAX_DECIMAL_PRECISION_SCALE - lprec +
> > lscale - rscale, 0);
> >         }
> > 
> > Here val is returning zero for scale, it seems.
> > 


Mime
View raw message