db-derby-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mike Matrigali <mikem_...@sbcglobal.net>
Subject Re: [jira] Commented: (DERBY-47) Some possible improvements to IN optimization
Date Thu, 29 Mar 2007 21:22:22 GMT


Army wrote:
> Mike Matrigali wrote:
> 
>> If there are no query plans printed then maybe the right thing to do
>> is to consider running the test multiple times, with different 
>> properties set. 
> 
> 
> [snip]
> 
>> maybe we could configure all the non-queryplan printing tests into
>> a suite and then enable or disable the interesting optimizer defaults.
> 
> 
> I agree that these are good ideas in terms of testing thoroughness.  My 
> knee-jerk reaction, though, is that we could end up running a lot of 
> tests many times in order to get all of the "interesting" cases.  My 
> guess is that such an approach would significantly increase the running 
> time of suites.All, which is something I think we'd rather avoid 
> (another reason we're switching to JUnit is the hope that the suite will 
> run more quickly, I believe?).
I agree, I don't think we should run every test, under every 
configuration, in suites.ALL.  We should determine what is a reasonable
set of tests for nightly.  But it would be nicely to be able to easily
configure a one-off set of suites.ALL run under an alternate 
configuration.  So for instance it would be interesting to run all the
tests under the non-cost based optimizer once per release/per week/per 
???, but maybe
not so interesting to do it as part of a nightly checkin.
> 
> Maybe an alternate approach is to make sure that any tests which set 
> optimizer-specific properties run (and pass) with the "default" 
> optimizer configuration, as well, where "default" means no optimizer 
> properties are set. The exception here would of course be tests that 
> print query plans, in which case we'd either 1) exempt them from the 
> "default" run, or 2) do the "default" run but skip query plan validation 
> in that case, thereby just verifying that the queries complete without 
> error and return the correct results.

Sounds good, I could imagine whatever the routine is that checks that
a query plan passes could somehow be able to check if the configuration
it is running against supports reproduceable query plans and just not
check if it doesn't.


> 
> There are two advantages to this approach.  First, the number of tests 
> which explicitly set "derby.optimizer.*" properties is, I think, 
> relatively small compared to the total number of tests.  And if you 
> ignore the "noTimeout" property it's even smaller.  So the number of 
> tests that see repeat runs would be pretty minimal.  Second, my guess is 
> that most users of Derby do _not_ set any optimizer-specific properties 
> when they run their apps, in which case the "default" test case (setting 
> no optimizer properties) would be a (more) typical user environment, 
> which I think is good.  Of course I could be wrong about what typical 
> users do here...?
> 
I agree that we should emphasize the default case.  Off hand I can't 
think of why most of the tests are setting other flags other than
to get around spurious query plan/ordering diffs.  I would expect at
least one specific test for non-cost based optimizer, if one does not
exist we probably should create one.

Unless one spends the time while converting the test to really 
understand what is being tested then it may be safest to do as you
suggest.  Convert the test to junit with no properties (this may enhance
what we are currently testings) and also run
it under junit with the existing properties (which will mean we won't
lose any coverage by the test conversion).

> Also note that such an approach would have correctly found DERBY-2500.
> 
>> By converting this to JUNIT it should be much easier to deal with 
>> results that are not guaranteed to be a specific order by having a 
>> single shared routine that checks for ordered, unordered, and maybe also
>> unordered with possible inconsistent number of duplicates.  This is 
>> something that was cludged in current .sql tests but can be done right
>> in junit tests.
> 
> 
> I think Knut Anders just checked in an "assertUnorderedResultSet()" 
> method this morning for exactly that reason.  So we're moving in the 
> right direction...
> 
> Army
> 
> 
> 


Mime
View raw message