river-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Costers <jonathan.cost...@googlemail.com>
Subject Re: ServiceDiscoveryManager test coverage
Date Fri, 27 Aug 2010 00:09:29 GMT
That's strange ... Hudson lists a user "Peter Firmstone" with last activity
4 days ago.
The user that was setup for me last year unfortunately disappeared from the
list :-(

2010/8/27 Peter Firmstone <jini@zeus.net.au>

> Ok, sounds good, I don't have access to Hudson unfortunately though, can
> anyone assist here?
>
>
> Jonathan Costers wrote:
>
>> Ideally it should, yes. So we get visibility on failures sooner ...
>>
>> To accomplish that, we need to make a change to the property value of
>> run.categories and add the missing categories (I've already added two more
>> the last cpl days).
>> However, adding all other categories to the run.categories property will
>> make our next Hudson builds take a very, very long time.
>>
>> So, as proposed above, we could (request to) setup a separate Hudson job
>> that does this complete test run nightly, instead of triggered by SVN
>> changes.
>> - the current Hudson job would build triggered by SVN changes and run some
>> unit tests and/or some selected QA tests (fast)
>> - a new Hudson job would do nighly (or otherwise scheduled) builds and run
>> the whole QA suite (very slow)
>>
>> How does that sound to you?
>>
>> Speaking of Hudson, any chance we can get that up and running again? I see
>> nothing has been built in a while.
>>
>> 2010/8/27 Peter Firmstone <jini@zeus.net.au>
>>
>>
>>
>>> Ok, I'm happy if we do that then, I just want to make sure I'm running
>>> all
>>> the tests I should, that's all, so qa.run should run all categories by
>>> default?
>>>
>>> Peter.
>>>
>>>
>>> Jonathan Costers wrote:
>>>
>>>
>>>
>>>> There is one now already: qa.run
>>>> We just need to pass it more test category names (property
>>>> run.categories)
>>>> by default.
>>>> Maybe I am misunderstanding you?
>>>>
>>>> 2010/8/27 Peter Firmstone <jini@zeus.net.au>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>> Hi JC,
>>>>>
>>>>> Can we have an ant target for running all the tests?
>>>>>
>>>>> And how about a qa.run.hudson target?
>>>>>
>>>>> I usually use run-categories, to isolate what I'm working on, but we
>>>>> definitely need a target that runs everything that should be, even if
>>>>> it
>>>>> does take overnight.
>>>>>
>>>>> Regards,
>>>>>
>>>>> Peter.
>>>>>
>>>>>
>>>>> Jonathan Costers wrote:
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>> 2010/8/24 Patricia Shanahan <pats@acm.org>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>> On 8/22/2010 4:57 PM, Peter Firmstone wrote:
>>>>>>> ...
>>>>>>>
>>>>>>>  Thanks Patricia, that's very helpful, I'll figure it out where
I
>>>>>>> went
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> wrong this week, it really shows the importance of full test
>>>>>>>> coverage.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> ...
>>>>>>>
>>>>>>> I strongly agree that test coverage is important. Accordingly,
I've
>>>>>>> done
>>>>>>> some analysis of the "ant qa.run" output.
>>>>>>>
>>>>>>> There are 1059 test description (*.td) files that exist, and
are
>>>>>>> loaded
>>>>>>> at
>>>>>>> the start of "ant qa.run", but that do not seem to be run. I've
>>>>>>> extracted
>>>>>>> the top level categories from those files:
>>>>>>>
>>>>>>> constraint
>>>>>>> discoveryproviders_impl
>>>>>>> discoveryservice
>>>>>>> end2end
>>>>>>> eventmailbox
>>>>>>> export_spec
>>>>>>> io
>>>>>>> javaspace
>>>>>>> jeri
>>>>>>> joinmanager
>>>>>>> jrmp
>>>>>>> loader
>>>>>>> locatordiscovery
>>>>>>> lookupdiscovery
>>>>>>> lookupservice
>>>>>>> proxytrust
>>>>>>> reliability
>>>>>>> renewalmanager
>>>>>>> renewalservice
>>>>>>> scalability
>>>>>>> security
>>>>>>> start
>>>>>>> txnmanager
>>>>>>>
>>>>>>> I'm sure some of these tests are obsolete, duplicates of tests
in
>>>>>>> categories that are being run, or otherwise inappropriate, but
there
>>>>>>> does
>>>>>>> seem to be a rich vein of tests we could mine.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> The QA harness loads all .td files under the "spec" and "impl"
>>>>>> directories
>>>>>> when starting and only witholds the ones that are tagged with the
>>>>>> categories
>>>>>> that we specify from the Ant target.
>>>>>> Whenever a test is really obsolete or otherwise not supposed to run,
>>>>>> it
>>>>>> is
>>>>>> marked with a "SkipTestVerifier" in its .td file.
>>>>>> Most of these are genuine and should be run though.
>>>>>> There are more categories than the ones you mention above, for
>>>>>> instance:
>>>>>> "spec", "id", "id_spec", etc.
>>>>>> Also, some tests are tagged with multiple categories and as such
>>>>>> duplicates
>>>>>> can exist when assembling the list of tests to run.
>>>>>>
>>>>>> The reason not all of them are run (by Hudson) now is that we give
a
>>>>>> specific set of test categories that are known (to me) to run
>>>>>> smoothly.
>>>>>> There are many others that are not run (by default) because issue(s)
>>>>>> are
>>>>>> present with one or more of the tests in that category.
>>>>>>
>>>>>> I completely agree with the fact that we should not exclude complete
>>>>>> test
>>>>>> categories because of one test failing.
>>>>>> What we probably should do is tag any problematic test (due to
>>>>>> infrastructure or other reasons) with a SkipTestVerifier for the
time
>>>>>> being
>>>>>> so that it is not taken into account by the QA harness for now.
>>>>>> That way, we can add all test categories to the default Ant run.
>>>>>> However, this would take a large amount of time to run (I've tried
it
>>>>>> once,
>>>>>> and killed the process after several days), which brings us to your
>>>>>> next
>>>>>> point:
>>>>>>
>>>>>> Part of the problem may be time to run the tests. I'd like to propose
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>> splitting the tests into two sets:
>>>>>>>
>>>>>>> 1. A small set that one would run in addition to the relevant
tests,
>>>>>>> whenever making a small change. It should *not* be based on skipping
>>>>>>> complete categories, but on doing those tests from each category
that
>>>>>>> are
>>>>>>> most likely to detect regression, especially regression due to
>>>>>>> changes
>>>>>>> in
>>>>>>> other areas.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> Completely agree. However, most of the QA tests are not clear unit
or
>>>>>> regression tests. They are more integration/conformance tests that
>>>>>> test
>>>>>> the
>>>>>> requirements of the spec and its implementation.
>>>>>> Identifying the list of "right" tests to run as part of the small
set
>>>>>> you
>>>>>> mention would require going through all 1059 test descriptions and
>>>>>> their
>>>>>> sources.
>>>>>>
>>>>>> 2. A full test set that may take a lot longer. In many projects,
there
>>>>>> is
>>>>>> a
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>> "nightly build" and a test sequence that is run against that
build.
>>>>>>> That
>>>>>>> test sequence can take up to 24 hours to run, and should be as
>>>>>>> complete
>>>>>>> as
>>>>>>> possible. Does Apache have infrastructure to support this sort
of
>>>>>>> operation?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> Again, completely agree. I'm sure Apache supports this through Hudson.
>>>>>> We
>>>>>> could request to setup a second build job, doing nightly builds and
>>>>>> running
>>>>>> the whole test suite. Think this is the only way to make running
the
>>>>>> complete QA suite automatically practical.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Are there any tests that people *know* should not run? I'm thinking
>>>>>>> of
>>>>>>> running the lot just to see what happens, but knowing ones that
are
>>>>>>> not
>>>>>>> expected to work would help with result interpretation.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> See above, tests of that type should have already been tagged to
be
>>>>>> skipped
>>>>>> by the good people that donated this test suite.
>>>>>> I've noticed that usually, when a SkipTestVerifier is used in a .td
>>>>>> file,
>>>>>> someone has put some comments in there to explain why it was tagged
as
>>>>>> such.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Patricia
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message