harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Geir Magnusson Jr <g...@pobox.com>
Subject Re: Unit testing revisited
Date Mon, 27 Mar 2006 08:47:36 GMT


Anton Avtamonov wrote:
> On 3/27/06, Geir Magnusson Jr <geir@pobox.com> wrote:
> [SNIP]
>> I don't think it's easier to write test cases for internals though a
>> public API, because you can call the internal methods directly to see if
>> they do as they promise, where it's not so easy to always have the same
>> thing happen through an indirect public API.
>>
>> Also, wrt things breaking..... yes, tests that ensure that something
>> works as expected will need to change if the expectation changes.
>>
>> That also is exactly what you want, right?  To know if you broke a
>> 'internal contract' in a class?
> 
> Hmmm... Lemme support Richard's point.
> At the first, internals not just specify 'internal functional
> contracts', but design decisions. Making redesign you will probably
> have some (design-dependent) unit tests failing, despite functionality
> is not changed. If so we will have to change the tests, right? If we
> changing the tests we cannot be sure they still tests the original
> behavior. IMHO, tests are the formal and executable way to represent
> requirements (XP approach) so they should reflect the 'requirements
> level'. In our case that is public API.

1) We're not doing XP.  We actually have predefined requirements.

2) For the sake of argument, I'll grant you that tests are the formal 
and executable way to represent the requirements, but that would be the 
spec/API tests (which we need as well).

2) The test failing is the correct thing, because the functionality did 
change.

Suppose you made a change that you *thought* didn't change the 
functionality, but actually did.  Wouldn't you want a fine-grained unit 
test to find that early?

> 
> At the second, what we really need to satisfy is API behavior only.

In the end, yes.

> Internal implementation is not significantly important if public part
> works fine.

Mmmmm.  I'm not so sure.

> I fond of XP and test-driven development and don't feel
> that we need to test something which is not part of requirements :-).
> If something is not accessible from the usage scenarios of public API
> level, it is not important how it works (I believe this idea appeared
> several times already in this thread).

And I don't agree, since I don't believe that you can make a test suite 
w/ sufficient coverage.  The only real opportunity that we have is 
something like gump, which punts on the formal test approach, and does 
it a different way - by running anything and everything in a 
"monte-carlo"-like technique which hopes to cover the parameter space. 
IMO, Gump is a fantastic tool for testing cross-project API contracts, 
for example, and will be very helpful for assisting us in finding 
problems, but I still believe that a deliberate testing approach both at 
the implementation level as well as the spec level would serve us well.

> I completely understand the issue of complexity if we test big amout
> of code referenced by public API, however I believe that in the
> majority of cases the behavior of each API method is not very
> complicated and can be tested by few scenarios. At least I haven't
> seen spec for some method of several pages long :-) Usually the
> behavior is comparative simple.
> 
> At the end (but not least important) you have to 'verify' your tests
> somehow. You prepare them basing on specification, however you should
> check them on RI. Just to make sure test is correct, spec corresponds
> to RI, etc. I cannot understand how it could be done for
> 'internals'...

It's irrelevant for internals, because internals our our implementation. 
  No need to verify that our internals tests work on anything else.


geir

Mime
View raw message