river-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Costers <jonathan.cost...@googlemail.com>
Subject Re: ServiceDiscoveryManager test coverage
Date Thu, 26 Aug 2010 23:46:02 GMT
Ideally it should, yes. So we get visibility on failures sooner ...

To accomplish that, we need to make a change to the property value of
run.categories and add the missing categories (I've already added two more
the last cpl days).
However, adding all other categories to the run.categories property will
make our next Hudson builds take a very, very long time.

So, as proposed above, we could (request to) setup a separate Hudson job
that does this complete test run nightly, instead of triggered by SVN
changes.
- the current Hudson job would build triggered by SVN changes and run some
unit tests and/or some selected QA tests (fast)
- a new Hudson job would do nighly (or otherwise scheduled) builds and run
the whole QA suite (very slow)

How does that sound to you?

Speaking of Hudson, any chance we can get that up and running again? I see
nothing has been built in a while.

2010/8/27 Peter Firmstone <jini@zeus.net.au>

> Ok, I'm happy if we do that then, I just want to make sure I'm running all
> the tests I should, that's all, so qa.run should run all categories by
> default?
>
> Peter.
>
>
> Jonathan Costers wrote:
>
>> There is one now already: qa.run
>> We just need to pass it more test category names (property run.categories)
>> by default.
>> Maybe I am misunderstanding you?
>>
>> 2010/8/27 Peter Firmstone <jini@zeus.net.au>
>>
>>
>>
>>> Hi JC,
>>>
>>> Can we have an ant target for running all the tests?
>>>
>>> And how about a qa.run.hudson target?
>>>
>>> I usually use run-categories, to isolate what I'm working on, but we
>>> definitely need a target that runs everything that should be, even if it
>>> does take overnight.
>>>
>>> Regards,
>>>
>>> Peter.
>>>
>>>
>>> Jonathan Costers wrote:
>>>
>>>
>>>
>>>> 2010/8/24 Patricia Shanahan <pats@acm.org>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>> On 8/22/2010 4:57 PM, Peter Firmstone wrote:
>>>>> ...
>>>>>
>>>>>  Thanks Patricia, that's very helpful, I'll figure it out where I went
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>> wrong this week, it really shows the importance of full test coverage.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>> ...
>>>>>
>>>>> I strongly agree that test coverage is important. Accordingly, I've
>>>>> done
>>>>> some analysis of the "ant qa.run" output.
>>>>>
>>>>> There are 1059 test description (*.td) files that exist, and are loaded
>>>>> at
>>>>> the start of "ant qa.run", but that do not seem to be run. I've
>>>>> extracted
>>>>> the top level categories from those files:
>>>>>
>>>>> constraint
>>>>> discoveryproviders_impl
>>>>> discoveryservice
>>>>> end2end
>>>>> eventmailbox
>>>>> export_spec
>>>>> io
>>>>> javaspace
>>>>> jeri
>>>>> joinmanager
>>>>> jrmp
>>>>> loader
>>>>> locatordiscovery
>>>>> lookupdiscovery
>>>>> lookupservice
>>>>> proxytrust
>>>>> reliability
>>>>> renewalmanager
>>>>> renewalservice
>>>>> scalability
>>>>> security
>>>>> start
>>>>> txnmanager
>>>>>
>>>>> I'm sure some of these tests are obsolete, duplicates of tests in
>>>>> categories that are being run, or otherwise inappropriate, but there
>>>>> does
>>>>> seem to be a rich vein of tests we could mine.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>> The QA harness loads all .td files under the "spec" and "impl"
>>>> directories
>>>> when starting and only witholds the ones that are tagged with the
>>>> categories
>>>> that we specify from the Ant target.
>>>> Whenever a test is really obsolete or otherwise not supposed to run, it
>>>> is
>>>> marked with a "SkipTestVerifier" in its .td file.
>>>> Most of these are genuine and should be run though.
>>>> There are more categories than the ones you mention above, for instance:
>>>> "spec", "id", "id_spec", etc.
>>>> Also, some tests are tagged with multiple categories and as such
>>>> duplicates
>>>> can exist when assembling the list of tests to run.
>>>>
>>>> The reason not all of them are run (by Hudson) now is that we give a
>>>> specific set of test categories that are known (to me) to run smoothly.
>>>> There are many others that are not run (by default) because issue(s) are
>>>> present with one or more of the tests in that category.
>>>>
>>>> I completely agree with the fact that we should not exclude complete
>>>> test
>>>> categories because of one test failing.
>>>> What we probably should do is tag any problematic test (due to
>>>> infrastructure or other reasons) with a SkipTestVerifier for the time
>>>> being
>>>> so that it is not taken into account by the QA harness for now.
>>>> That way, we can add all test categories to the default Ant run.
>>>> However, this would take a large amount of time to run (I've tried it
>>>> once,
>>>> and killed the process after several days), which brings us to your next
>>>> point:
>>>>
>>>> Part of the problem may be time to run the tests. I'd like to propose
>>>>
>>>>
>>>>
>>>>
>>>>> splitting the tests into two sets:
>>>>>
>>>>> 1. A small set that one would run in addition to the relevant tests,
>>>>> whenever making a small change. It should *not* be based on skipping
>>>>> complete categories, but on doing those tests from each category that
>>>>> are
>>>>> most likely to detect regression, especially regression due to changes
>>>>> in
>>>>> other areas.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>> Completely agree. However, most of the QA tests are not clear unit or
>>>> regression tests. They are more integration/conformance tests that test
>>>> the
>>>> requirements of the spec and its implementation.
>>>> Identifying the list of "right" tests to run as part of the small set
>>>> you
>>>> mention would require going through all 1059 test descriptions and their
>>>> sources.
>>>>
>>>> 2. A full test set that may take a lot longer. In many projects, there
>>>> is
>>>> a
>>>>
>>>>
>>>>
>>>>
>>>>> "nightly build" and a test sequence that is run against that build.
>>>>> That
>>>>> test sequence can take up to 24 hours to run, and should be as complete
>>>>> as
>>>>> possible. Does Apache have infrastructure to support this sort of
>>>>> operation?
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>> Again, completely agree. I'm sure Apache supports this through Hudson.
>>>> We
>>>> could request to setup a second build job, doing nightly builds and
>>>> running
>>>> the whole test suite. Think this is the only way to make running the
>>>> complete QA suite automatically practical.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>> Are there any tests that people *know* should not run? I'm thinking of
>>>>> running the lot just to see what happens, but knowing ones that are not
>>>>> expected to work would help with result interpretation.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>> See above, tests of that type should have already been tagged to be
>>>> skipped
>>>> by the good people that donated this test suite.
>>>> I've noticed that usually, when a SkipTestVerifier is used in a .td
>>>> file,
>>>> someone has put some comments in there to explain why it was tagged as
>>>> such.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>> Patricia
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message