db-derby-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ole Solberg <Ole.Solb...@Sun.COM>
Subject Re: how best to describe regression test bugs to enable the "automatic" tool in the sun test suite to work?
Date Tue, 26 Sep 2006 11:38:07 GMT
Mike Matrigali wrote:
> Can anyone describe the best way to report regression test failures so
> that the tool that produces the following report will pick it up:
> 
> http://www.multinet.no/~solberg/public/Apache/Failures/TinderBox-10.1/448426.html
> 
> 
> For this issue I reported DERBY-1871 for the specific diff, but I
> probably have not described it in a way the tool can find.  I would be
> happy to update the jira description, but just looking for some hints
> on what it is looking for.
> 

Currently I only maintain a file which maps the name of the failing test
method (in this case 'derbynet/testSecMec.java') to the possible JIRA.
So for this case I would add 1871 as another JIRA which may be caused by
'derbynet/testSecMec.java' failing.
If we could have the failing test method as part of the JIRA
description, or in some special field, that would help keep the mapping
file up to date.


But, as there may be a many-to-many mapping between the test method
failing and JIRA, we need a better "signature": We now often get a list
of possible JIRAs per test method.

What I am in the process of doing is to try to map the "real" signature,
that is e.g. the diff produced by the derby test harness, to the JIRA
number. This should give a higher confidence in what JIRA we are
actually seeing.
I will experiment with setting up a Derby database to consult for these
mappings.

If JIRA contained a field with the "signature", we might even be able to
periodically update this "bug" database with new JIRA-"signature"
mappings from JIRA. (This we could do even with todays test method as
"signature".)


-- 
Ole Solberg, Database Technology Group,
Sun Microsystems, Trondheim, Norway

Mime
View raw message