db-derby-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew McIntyre <mcintyr...@gmail.com>
Subject Re: Regression testing
Date Wed, 11 May 2005 09:18:03 GMT

On May 10, 2005, at 11:00 AM, David Van Couvering wrote:

> Again, I'm not sure why there would be "lots of emails" for a nightly 
> build.  Also, if we email failures, that would be great, as I know I 
> personally won't be checking the test web site on a regular basis.

Let's get hypothetical and say 2 derby-dev'rs decide to run the tests 
nightly against their own builds (on whatever platform/jdk revision 
they're interested in) and send the output to the list. Now, make that 
5. or 10. Imagine if, by accident, someone checks in a change that 
happens to breaks all of the tests, and we have 5 people sending their 
test result failures to the list. Suddenly, 300 people have 100 
megabytes of test diffs in their inbox. Not fun.

But, what if instead derby-dev'r X running tests on platform Y and JDK 
Z posts to the list with "there is a serious failure going on, I think 
it's due to {whatever} and here's a link to the failure information: 
.... " - that's a lot more useful then automatic notification, even in 
the case of failure. And, with a page on the website that can collect 
the locations of where specific platform/jvm tests are running, at 
least all of that useful test information is available without 
derby-dev receiving a lot of email that is only of interest to a small 
subsection of people. If there are serious issues on a specific 
platform/jvm, then the people responsible for those tests can provide a 
more detailed description and bug report to derby-dev than if a failure 
notification is mailed to the list.  Well, that's my opinion, anyway

> Well, it seems to me you just have a cycle of "pull, build, full 
> regresssions, post results."  If we have a powerful enough machine, it 
> shouldn't take that long, and we'd have fresh results every, say, four 
> hours.  Not bad!
>
> If we want a faster turnaround, we could, as you suggest, identify a 
> subset of tests.

I don't think four hours for a full build/test cycle is unreasonable 
for a tinderbox, and I think with fairly recent hardware and the 
relaxed durability option for testing, that the time for a full 
build/test cycle could probably be quite a bit less than that, probably 
around two hours on top-of-the-line hardware. i.e. I think that running 
the full test suite in a tinderbox approach is a better idea than a 
subset of the tests.

andrew


Mime
View raw message