harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Geir Magnusson Jr <g...@pobox.com>
Subject Re: regression test suite
Date Fri, 30 Dec 2005 16:29:41 GMT


Tim Ellison wrote:
> Geir Magnusson Jr wrote:
> 
>>Thanks for bringing this up.  This has been on my to-do list to talk about.
>>
>>inline
>>
>>Vladimir Strigun wrote:
>>
>>
>>>We already discussed several questions about tests. But I'd like to
>>>open a new topic about regression test suite. Some API bugs have been
>>>fixed (or patch suggested) and I think it will be useful to have
>>>regression tests for all these bugs to avoid them reappearing in the
>>>future.
>>>
>>>I think the natural place for the regression tests should be together
>>>with the unit tests. I am not sure, however, if the regression tests
>>>should be marked explicitly or differ from the unit tests in any other
>>>specific way.
>>>
>>>What do you think about it? Any preferences?
>>
>>
>>I think that putting them in a parallel tree is worth considering, just
>>for the sake of organization, as long as this is transparent to the
>>tools a developer/user would use to do run the tests.  I do think that
>>"regression test" is a broad term, and we might want to harvest things
>>out of regression tests for where our unit tests fell short.
> 
> 
> Aren't we getting a bit ahead of ourselves here, the existing tests
> hardly need organizing (all three of them!) -- but let's go mad and
> assume we have lots of tests housed in our repository.

Not really, as there are quite a few in the intel security contribution. 
  (3?  That's pathetic!)

> 
> What is the useful distinction for regression tests being kept separate?
>  I can see that you may distinguish unit and 'system-level' tests just
> because of the difference in frameworks etc. required, but why do I care
> if the test was written due to a JIRA issue or test-based development or
> someone who get's kicks out of writing tests to break the code?

Well, it's been my experience that unit tests generally are fairly 
straightforward and isolated, where a regression test used to 
demonstrate a bug might be fairly complicated, incorporating more than 
the single unit you are trying to test/fix/debug/whatever.

> 
> 
>>I also believe we should be encourging bug reporters to submit a test
>>that demonstrates the bug.  We'd acquire a good set of tests at that point.
> 
> 
> Agreed, a simple failing test with a description of expected results is
> the ideal way to report a bug.  In particular, people are not expected
> to suggest a patch when they raise issues, though they are of course
> welcome to do so.

s/welcome/strongly encouraged/ :)

> 
> 
>>I was going to take a hard look at the build/test framework this
>>week(end).  Might as well start now.  Currently, it's JUnit.  I'm
>>wondering if anyone would consider switching to TestNG.  I've never used
>>it, but it's written by a guy I really respect and the point was to
>>address shortcomings he found in JUnit...
> 
> 
> I've never used TestNG, so have no opinion at the moment.  What's so
> good about it?

I'll report back.

Since we have no current setup for tests, I'll assume some leeway to 
take a run at it as there are no existing toes to step on.  First task 
will be to get your 3 [pathetic! :) ] unit tests running

geir

> Regards,
> Tim
> 
> 
>>geir
>>
>>
>>>Thank you,
>>>Vladimir Strigun, Intel Middleware Products Division.
>>>
>>>
>>
> 

Mime
View raw message