db-derby-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rick Hillegas <Richard.Hille...@Sun.COM>
Subject Re: [jira] Updated: (DERBY-937) Instability in wisconsin test
Date Fri, 30 Jun 2006 16:17:51 GMT
Mike Matrigali wrote:

>
>
> Oystein Grovlen - Sun Norway wrote:
>
>>
>> While it is good that we get this noise out of derbyall, it seems 
>> like this fix covers up a problem that we do not quite understand.  
>> As far as  I have understood, it seems like the optimizer due to some 
>> timing issues does not use the indexes.  I think we should file a 
>> JIRA for this problem.
>>
>
> I was working under the assumption that the wisc diffs were the kinds of
> issues we had seen in the past.  Where the optimizer had 2 different 
> plans which were very close in costing, and due to slight differences in
> estimates one plan was chosen over another.
>
> If anyone has done the careful analysis of the different plans and 
> found that there is a serious problem with the plans the optimizer was 
> picking please go ahead and report a new bug with that info and how to 
> reproduce.
>
> My main concern was that spurious "expected" diffs means that almost 
> everyone is just going to ignore any real future bug the test might
> show up.
>
I agree with what Mike said. I have worked for several database 
companies and at all of these companies, optimizer choices were 
notoriously subject to machine characteristics and therefore optimizer 
output was problematic for regression tests. I have seen this issue 
handled three ways:

1) An optimizer expert volunteers to monitor the optimizer drift in the 
nightly regression tests and judge whether the drift is signal or noise.
2) The tests are re-coded to check for something less noisy than 
optimizer output.
3) The tests are removed from the standard regression suite and only run 
at the discretion of the optimizer experts.

Mike is right: If these tests reliably generate noise, they will be 
ignored. This is pretty much equivalent to option (3).


Mime
View raw message