db-derby-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Van Couvering <David.Vancouver...@Sun.COM>
Subject Re: Regression testing
Date Wed, 11 May 2005 18:10:36 GMT

Andrew McIntyre wrote:

> On May 10, 2005, at 11:00 AM, David Van Couvering wrote:
>> Again, I'm not sure why there would be "lots of emails" for a nightly 
>> build.  Also, if we email failures, that would be great, as I know I 
>> personally won't be checking the test web site on a regular basis.
> Let's get hypothetical and say 2 derby-dev'rs decide to run the tests 
> nightly against their own builds (on whatever platform/jdk revision 
> they're interested in) and send the output to the list. Now, make that 
> 5. or 10. Imagine if, by accident, someone checks in a change that 
> happens to breaks all of the tests, and we have 5 people sending their 
> test result failures to the list. Suddenly, 300 people have 100 
> megabytes of test diffs in their inbox. Not fun.

Oh, OK, I wasn't thinking about it that way.  I was thinking there is 
one official nightly run of regression tests, and a report is sent out 
by email.  This report would include all covered platforms.  I wasn't 
thinking of this as a distributed job.

But you're probably right, it makes more sense to distribute the work, 
and have separate derby-devrs to choose the platforms they are 
interested in, and have a common web site where the results are posted...

> But, what if instead derby-dev'r X running tests on platform Y and JDK Z 
> posts to the list with "there is a serious failure going on, I think 
> it's due to {whatever} and here's a link to the failure information: 
> .... " - that's a lot more useful then automatic notification, even in 
> the case of failure. And, with a page on the website that can collect 
> the locations of where specific platform/jvm tests are running, at least 
> all of that useful test information is available without derby-dev 
> receiving a lot of email that is only of interest to a small subsection 
> of people. If there are serious issues on a specific platform/jvm, then 
> the people responsible for those tests can provide a more detailed 
> description and bug report to derby-dev than if a failure notification 
> is mailed to the list.  Well, that's my opinion, anyway
>> Well, it seems to me you just have a cycle of "pull, build, full 
>> regresssions, post results."  If we have a powerful enough machine, it 
>> shouldn't take that long, and we'd have fresh results every, say, four 
>> hours.  Not bad!
>> If we want a faster turnaround, we could, as you suggest, identify a 
>> subset of tests.
> I don't think four hours for a full build/test cycle is unreasonable for 
> a tinderbox, and I think with fairly recent hardware and the relaxed 
> durability option for testing, that the time for a full build/test cycle 
> could probably be quite a bit less than that, probably around two hours 
> on top-of-the-line hardware. i.e. I think that running the full test 
> suite in a tinderbox approach is a better idea than a subset of the tests.

OK, I think we're in agreement here then...  Now it's just a mere matter 
of implementing it :)


> andrew

View raw message