river-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Patricia Shanahan <p...@acm.org>
Subject Re: Open discussion on development process
Date Sun, 02 Dec 2012 18:29:21 GMT
I hope you don't mind me throwing in some random comments on this. I 
think there are two types of testing that need to be distinguished, 
system and unit.

A system test looks that the external behavior of the whole system, so 
what it is testing changes only when the API changes, and tests should 
apply across many source code revisions. I can see separating those out.

However, I feel River has been weak on unit tests, tests that check the 
implementation of e.g. a data structure against its javadoc comments. 
Those tests need to change with internal interface changes.

Testing e.g. the multi-thread consistency of a complex data structure 
using only external system tests can be a mistake. It may take a very 
large configuration or many days of running to bring out a bug that 
could be found relatively fast by a unit test that hits the data 
structure rapidly.

I'm a little concerned that the reconfiguration of the tests may 
represent an increased commitment to only doing system tests, and not 
doing any unit tests.

Patricia


On 12/2/2012 10:21 AM, Dan Creswell wrote:
> ...
>
> On 30 November 2012 19:53, Gregg Wonderly <gergg@cox.net> wrote:
>> I still wonder why it doesn't feel right that the test suite be in the same branch
as the associated "release".  Some of the new code needs new test that demonstrate "functionality"
while other tests that demonstrate compatibility will be ran on each release without change.
 It seems to me, that in the end, when a release goes out the door, the tests that validated
that release, are part of that "release".
>>
>
> I have some similar disquiet, here's what I'm thinking at the moment
> (subject to change faster than I can type!)...
>
> Compatibility and similar is really "compliance test" and is closely
> linked to the APIs defined by the specs. Two flavours here:
>
> (1) "Well-behaved service" tests - does a service do join properly etc.
> (2) Compliance tests - do the APIs behave right etc.
>
> These are kind of slow moving as are the APIs at least for now. I feel
> right now like (1) might be a subproject applied to our own "built-in"
> services as well as others. I'm tempted to say the same about (2) save
> for the fact that if we give up on the idea someone else is going to
> build a River clone this stuff becomes part of the release/test phase
> for the core.
>
> Any other testing we're doing over and above what falls into (1) and
> (2) above is part of tests for core and ought to be living in the same
> branch and run as part of release. However, that's a little
> uncomfortable when one wishes to freeze development of core to do
> major work on the test harness etc. You branch core and test suite to
> work purely on the suite.
>
> Manageable I guess well, until you have the trunk moving on and
> breaking your already seriously under construction test suite where
> everything in trunk is "old style" and will be a b*stard to merge
> across but if you don't your branched test suite is gonna break for
> nuisance reasons.
>
>> If we need two different types of tests, and a migration path from "functionality
tests" into "compatibility tests", then maybe we really need two trees for development of
each release, and branching the whole test suite would be one branch an the new release would
be the other.
>>
>> Is that how you guys are thinking about this?
>>
>
> You have my (current) thinking above...
>
>> Gregg Wonderly
>>
>> On Nov 30, 2012, at 9:43 PM, Peter Firmstone <jini@zeus.net.au> wrote:
>>
>>> On 30/11/2012 12:27 AM, Dan Creswell wrote:
>>>> On 29 November 2012 13:11, Peter Firmstone<jini@zeus.net.au>  wrote:
>>>>
>>>>> The last passing trunk versions:
>>>>>
>>>>> Jdk6 Ubuntu     1407017
>>>>> Solaris  x86        1373770
>>>>> Jdk7 Ubuntu     1379873
>>>>> Windows           1373770
>>>>>
>>>>> Revision 1373770 looks the most stable, I think all platforms were passing
>>>>> on this,  1407017 only passed on Ubuntu jdk6, nothing else.
>>>>>
>>>>> If we can confirm 1373770 as stable, maybe we should branch a release
off
>>>>> that, buying some time to stabilise what we're working on now.
>>>>>
>>>>>
>>>> I think we should do that. I'm also tempted to suggest we consider limiting
>>>> our development until we've fixed these tests up. Or alternatively control
>>>> the rate of patch merging so we can pace it and make sure the tests get
>>>> focus.
>>>>
>>>> That's a bit sledgehammer but...
>>>>
>>>>
>>>
>>> Ok, sounds like a plan, how do you think we should best approach the task?
>>>
>>> Create a branch in skunk, just for qa and run tests against released jars?
>>>
>>> Regards,
>>>
>>> Peter.
>>>
>>>
>>


Mime
View raw message