On 2/2/06, Andreas Korneliussen <Andreas.Korneliussen@sun.com > wrote:
I think the work currently done on DERBY-874 was mainly to improve the
DerbyJUnitTest's JavaDoc, and to log exceptions. So I would not throw
that away.

However I do propose to change DerbyJUnitTest to move out everything
about configuration into a separate class.
 
cool. thx for the reply.
 
I now noticed that the wiki says all suggestions are to be put on the list, so here I go rather than plopping them directly on the wiki:
 
I think the following could qualify as 'more details' to the jvm, framework, version specific logic:
 
1. jvm-specific:
1.1.
not all parameters are consistent for all jvms. Think here of jit settings / configurations, memory settings. For j2ME testing, that jvm doesn't come with a DriverManager implementation, so already from the start you know you have to go with DataSources.
1.2. Different versions of a vendor's jvm also may have slightly different implementations resulting in slightly different behavior - e.g. of the order of rows, for instance, or rounding of certain numeric values.
1.3. Some behavior will be only supported by later versions...
 
2. version specific.
This really falls back to the discussion on ...(can't find right now, raman's working on it, I think)... re mixed version testing. I think the conclusion was that the harness needs a way to cope with results from newer servers and clients - if they differ from results with same versions as the harness.
 
3. framework specific
The tests needs to be able to cope with the following
3.1. different client drivers (e.g. DerbyClient, IBM Universal JDBC Driver)
3.2. server may need to be started by harness, or not
3.3. server may be on the localhost, or be running on a remote machine.
         certain individual tests may not be able to run in with this mechanism...
3.4 should be able to have the harness start the server in a differrent jvm.
 
4. one thing the current harness has no way of doing is to cope with different OSs. For instance, sometimes there are slight differences in behaviour of the same jvm version on different OSs. Like slightly different error messages (although this may be irrelevant if we're not gathering & comparing output).
 
I think the following details would be useful (in addition to the above and item 1 on the wiki):
- there must be a way to skip individual tests without causing an error but with an informational message for certain configurations. eg. absence of optional jars (specifically thinking of db2jcc.jar), unsupported functionality with older jvms..., or when there is a problem that's being worked on, or that's been referred to some other organization ( e.g. in the case of jvm bugs, OS bugs...).
 
- some way to compare runtimestatistics.
   Currently this is done by comparing the output, I have a hard time thinking of another mechanism.
 
Ok, that'll do for now...:-)
 
Myrna