Debugging test failures with the Derby test framework

Steps to be followed

1. Frist the test/s have to be run. The details fro running the tests can be found at ${derby.source}/java/testing/README.htm. The command for running the test/s would something like this,

java -Dij.exceptionTrace=true -Dkeepfiles=true org.apache.derbyTesting.functionTests.harness.RunTest lang/"mytest".sql

For the netwrok server you would need to add the following property -Dframework=DerbyNet

2. Do a visual diff of the ouptut with the canon. It will give you more context in approaching the problem. In the test output directory you will see "mytest".out (filitered test output) and "mytest".tmp (unfiltered test output - this one should have a trace). To get the most information, it is advised to diff the tmp file with the canon which is checked in under java/testing/org/apache/derbyTesting/functionTests/master/ or appropriate framework or jdk subdirectory.

3. Identify the sql statement or java code causing the diff. For sql scripts this is usually pretty obvious. For java programs you have to look at the test source to figure out what is going on.

4. Evaluate the diff. Here of course starts the tricky and interesting part. Look carefully at the sql command or java code that caused the diff. Think about what it should do and how it relates to your change. Decide what you think the right behaviour should be. a) The old behaviour b) The new behaviour c) something else If you have a trace, look at the the trace and see if it holds any clue. If you are lucky it passes right through that code you changed. Often it is helpful to put that one sql command or bit of java code in a separate script or stand alone program, so you can evaluate it independently outside of the harness and evaluate in the debugger. Common results of the evaluation phase are: a) An error in your code. b) Someone elses test failure all together. It is good to keep a clean client for testing this. c) A master update. Be careful with this one. make sure you have a valid reason to update a master. Be especially cognizant of backward compatibility. We don't want any real or perceived regressions to catch us by surprise.

Two good questions to ask your self and then answer when you post your patch. 1) What is my valid reason for updating this master. 2) Might someone be surprised by this difference and perceive it as a regression?

5. Resolve the issue. Here are some possible actions based on your evaluation in step 4 a) An error in your code. Go fix it !!! b) Someone else's test failure all together. Look at the recent checkins and try to guess the culprit and post. c) A master update. Update the master and make sure you include an explanation in your patch.

6. Reporting your findings. If you get stuck along the way, please post to the list, but make sure you include. a) The small bit of sql or java code in question. b) A description of the old and new behaviour and what you think the right behavior should be. c) The stack trace if there is one. d) What you have learned so far from your own evaluation of 1-3 above. e) A specific question that is not going to take a lot of research for someone to answer that you think might send you back along your way if answered.