river-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Patricia Shanahan <p...@acm.org>
Subject Re: Status of QA test verification
Date Wed, 15 Sep 2010 14:01:23 GMT
Jonathan Costers wrote:
> This is what we have running in Hudson now:
> 
> [java] -----------------------------------------
> [java]
> [java] # of tests started   = 750
> [java] # of tests completed = 750
> [java] # of tests skipped   = 21
> [java] # of tests passed    = 748
> [java] # of tests failed    = 2
> [java]
> [java] -----------------------------------------
> 
> Total time: 598 minutes 7 seconds
> 
> That's almost 10 hours ... :-) And we seem to have 50% coverage now (seeing
> there are about 1500 .td files available)

Assuming approximate linearity, the full set would take 20 hours to run. 
It is practical to do a 20 hour test once a day, but not once every few 
builds, and I like to test early, test often.

Perhaps we should create a new category, orthogonal to the existing 
categories, that selects about a one hour regression test that one could 
run regularly during development? I don't think dropping complete 
existing categories is a good way to do it, because that leads to 
complete non-testing of significant components.

Obviously, tests that have failed during the test expansion work are 
very strong candidates for inclusion in the reduced set - they test 
things that are affected by the changes you have backed out, and 
presumably re-implementing those changes is going to be an important 
part of the future work.

Incidentally, I've suspended my attempts to write more tests, and 
switched to TaskManager work, because I realized I don't know whether we 
have the tests I think are needed or not. I did know that I was not 
satisfied with the set that were being run, but that is a different matter.

Many, many thanks for doing this work on getting more tests running. I 
think it is extremely important.

Patricia

Mime
View raw message