river-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peter Firmstone <j...@zeus.net.au>
Subject Re: Open discussion on development process
Date Thu, 29 Nov 2012 11:34:34 GMT
On 29/11/2012 12:49 AM, Dan Creswell wrote:
>> It would be nice to have a stable qa suite branch for testing River
>> development and another for refactoring and developing the test suite
>> itself, without interfering with the development process of River.  The
>> test suite requires maintenance too, but right now it's a frightening
>> prospect.
>>
>>
> I think it's essential to sort this out. I reckon we ultimately should be
> aiming at having an up-to-date/parallel developed test suite for trunk and
> indeed skunk. Bad test/compile process otherwise.
>
> But there's a catch...
>
>
>>   I'd like to be able to run an experimental test suite with a stable River
>> trunk to test the test suite so to speak.
>>
>>
> I think that's wise equally I'm pondering how you cope with trunk moving on
> and breaking a bunch of your tests for which there would be fixes in the
> trunk test suite. Are you happy doing merges as and when or ???
>
>

I'm not sure, at this point I just know we've got problems, there's a 
lot of coupling in the test suite and a lot of shared state; it's all 
based on inheritance.

There are two test types:

   1. Jini Standards Compliance
   2. Implementation integration tests.

Then there's the harness itself.

I'm not sure how I'd handle trunk moving on, that could be a problem, it 
may be acceptable for implementation tests, but not so for Jini 
Standards Compliance.

Oh, BTW, when I turned off debugging I had 6 tests fail, then after 
running again only 2 tests failed; my recent patches haven't solved 
concurrency issues, debugging masked it.

In theory the implementation tests belong in trunk and could utilise a 
qa harness binary distribution, similar to how we use junit or jtreg, 
but we'd need to decouple the harness to achieve this.

The Jini Standards Tests should probably be bundled separately in a TCK.

Refactoring the harness won't be an easy task, the quality of trunk 
depends on it and we're going to experience more concurrency headaches 
until we do.

A starting point might be to copy the qa suite and tests to a skunk 
directory and run the harness against River release 2.2.0 and earlier, 
while refactoring.

We'd need to look at the services the harness provides and how to best 
provide them, perhaps using annotations, we could potentially make 
integration tests much simpler.

Regards,

Peter.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message