harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Geir Magnusson Jr <g...@pobox.com>
Subject Re: Test suite layout
Date Mon, 16 Jan 2006 13:45:45 GMT


George Harley1 wrote:
> Hi Geir, 
> 
> Reading "The Da Vinci Code" is common practice too. Doesn't make it a good 
> book. 

I think there is a difference between popular fiction and somewhat 
established engineering practice.

> 
> I wasn't trying to say that it is a bad practice in every case, just that 
> it does have its downsides and for that reason the assertion that "unit 
> test code must reside in the same package as the classes" is not correct. 
> It is possible to arrive at a high quality set of unit tests for a class 
> where the package names differ. 

Of course, although I'd be interested to see how you do fine-grain 
testing of the internals, which in my opinion is very useful, as the 
promises made by private and package access code to other class-level 
and package-level code is just as important to the promises made by 
public APIs to users.

> 
> In the interests of avoiding any religious war can I just say that there 
> is no religious war here : both unit test approaches work but neither 
> should we look on one as being "the only way". So let's not argue about it 
> on the list. Instead, let's see how the unit test code evolves over time 
> before drawing any conclusions. 

Right - feel free to comment on the other message, titled "Test 
Framework".  The initial discussion surrounding same-package unit tests 
stemmed from having to run on the boot classpath, IMO an important 
consideration.  However...

geir
> 
> Best regards, 
> George
> ________________________________________
> George C. Harley
> 
> 
> 
> 
> Geir Magnusson Jr <geir@pobox.com> 
> 16/01/2006 12:54
> Please respond to
> harmony-dev@incubator.apache.org
> 
> 
> To
> harmony-dev@incubator.apache.org
> cc
> 
> Subject
> Re: Test suite layout
> 
> 
> 
> 
> 
> 
> 
> 
> George Harley1 wrote:
>> Hi, 
>>
>>> This is not the goal of unit tests. They do not test API, they test the 
> 
>> code,
>>> including package-access members etc. That is why unit tests must 
> reside
>>> in the same package as the classes.
>> This is radical stuff. I have never heard the argument before that unit 
>> tests *must* share the same package name as the type under test. One 
>> important reason why many think it is a bad idea to take this approach 
> is 
>> that the resulting unit test code becomes too closely coupled with the 
>> code under test.
> 
> Huh?  It's a common practice - it gives the unit test the ability to 
> really test the implementation because it has package level priv.
> 
>> Think about it ... 
>>
>> If unit test code can access every package-access member and method in a 
> 
>> given type then the test code will eventually reach a point where it 
>> contains a lot of brittle assumptions about internal implementation. 
> 
> It can, yes.  OTOH, it is testing internal private methods, so yes, 
> there's going to be those assumptions built in.
> 
>> Consequently, when implementation changes occur, the corresponding test 
>> code will break. Note : I'm not talking about API changes breaking the 
>> test code here, I'm talking about changes to protected and 
> package-access 
>> stuff which - by definition - the developer has chosen to keep 
> non-public. 
> 
> Right - and the unit tests test that code.  How else do you test it in a 
> fine-grained way?  Only via public methods?
> 
> 
>> So as the internals change the unit tests break. And further internal 
>> changes make the unit tests break. And ultimately maintaining the tests 
>> with close-coupling to the internals of the type under test becomes a 
> bit 
>> of a pain and their overall value in the test harness diminishes because 
> 
>> minor refactorings to the type-under-test now take so darned long to 
> carry 
>> out. 
> 
> But it gives you very fine-grained coverage, if you want that.
> 
> geir
> 
>>
>> Best regards, 
>> George
>> ________________________________________
>> George C. Harley
>>
>>
>>
>>
>>
>> Mikhail Loenko <mloenko@gmail.com> 
>> 16/01/2006 11:58
>> Please respond to
>> harmony-dev@incubator.apache.org
>>
>>
>> To
>> harmony-dev@incubator.apache.org
>> cc
>>
>> Subject
>> Re: Test suite layout
>>
>>
>>
>>
>>
>>
>> On 1/16/06, Tim Ellison <t.p.ellison@gmail.com> wrote:
>> ...
>>> If your unit tests are intended to test API, then they should be 
> calling
>>> the API in the same manner that an application will call the API.  Just
>> This is not the goal of unit tests. They do not test API, they test the 
>> code,
>> including package-access members etc. That is why unit tests must reside
>> in the same package as the classes.
>>
>> Other kinds of tests like compatibility or functional tests call just
>> public and protected methods so they can be in different packages.
>>
>> Thanks,
>> Mikhail Loenko
>> Intel Middleware Products Division
>>
>>
>>
>>
>>
>>> look at the number of places where we check to see if the classloader 
> ==
>>> null to see where this matters.
>>>
>>>> Some of them may depend on security manager component if
>>>> you like to run a test in some security restricted environment. But it
>>>> is not a big issue: the security manager component can be configured
>>>> dynamically and you should install your custom security manager before
>>>> running a test.
>>> Yes, we should do that on the application classloader to test security
>>> manager functionality.
>>>
>>> Regards,
>>> Tim
>>>
>>>> What do you think?
>>>>
>>>> Thanks,
>>>> Stepan Mishura
>>>> Intel Middleware Products Division
>>>>
>>>>
>>>>> -----Original Message-----
>>>>> From: Tim Ellison [mailto:t.p.ellison@gmail.com]
>>>>> Sent: Thursday, January 12, 2006 8:04 PM
>>>>> To: harmony-dev@incubator.apache.org
>>>>> Subject: Re: Test suite layout
>>>>>
>>>>> Geir Magnusson Jr wrote:
>>>>>
>>>>>> Tim Ellison wrote:
>>>>> <snip>
>>>>>
>>>>>>> We would have written it as java.io.tests, but the java.<whatever>
>>>>>>> namespace is reserved, so the formula is simply
>>>>>>>
>>>>>>> <package>.<type>  -> org.apache.harmony.tests.<package>.<type>Test
>>>>>> Ug - then you have the problem of not being in the namespace as what
>>>> you
>>>>
>>>>>> are testing.
>>>>>>
>>>>>> THat's why people use parallel trees - so your test code is
>>>> physically
>>>>
>>>>>> separate but you have freedom of package access.
>>>>> For 'normal' application code then you can do this, but since we are
>>>>> writing the java packages themselves then you come unstuck because 
> the
>>>>> java packages have to run on the bootclasspath, and the tests on the
>>>>> application classpath.
>>>>>
>>>>> You don't want to run all your tests on the bootclasspath (because 
>> then
>>>>> it would not be subject to the same security sandboxing as 
>> applications
>>>>> using the APIs you are testing); and you cannot put java.<whatever>

> on
>>>>> the application classpath because the VM will catch you out (you'd 
> get
>>>> a
>>>>
>>>>> linkage error IIRC).
>>>>>
>>>>>
>>>>>>> This makes it clear what is being tested, and where to add new

> tests
>>>>> etc.
>>>>>
>>>>>> So would
>>>>>>
>>>>>>  test/org/apache/harmony/io/TestFoo.java
>>>>>>
>>>>>> (to test something in org.apache.harmony.io, and arguable to test

> the
>>>>>> Foo.java class in there.  (or TestFoo.java - it's early - no coffee
>>>> yet)
>>>>
>>>>> Not sure what you are saying here... For java.<whatever> packages
we
>>>>> need a prefix on the test packages to keep the VM happy, but for
>>>>> org.apache.harmony packages we can have either pre- or post-.
>>>>>
>>>>> I'd actually prefer a postfix of .tests for non-API packages, though

> I
>>>>> can understand if people object to the inconsistency; so
>>>>>
>>>>> org.apache.harmony.tests.java.io.FileTest.java      <- test API
>>>>> org.apache.harmony.io.tests.FileImpltest.java  <- test public methods
>>>>>                                                 in our IO impl'
>>>>>
>>>>>
>>>>>> Simiarly
>>>>>>
>>>>>>  test/java/util/TestMap.java
>>>>>>
>>>>>>
>>>>>>> Then within the test class itself the methods are named after
the
>>>> method
>>>>
>>>>>>> under test, with a familar JNI-style encoding,  so we have things
>>>> like:
>>>>
>>>>>>> org.apache.harmony.tests.java.io.FileTest contains
>>>>>>>    public void test_ConstructorLjava_io_FileLjava_lang_String()
{
>>>>>>>    ...
>>>>>>>    }
>>>>>>>
>>>>>>> and
>>>>>>>
>>>>>>> org.apache.harmony.tests.java.lang.FloatTest contains
>>>>>>>    public void test_compareToLjava_lang_Float() {
>>>>>>>    ...
>>>>>>>    }
>>>>>> ...or whatever the convention is for JUnit.  I think that's one of
>>>> the
>>>>
>>>>>> nice things about TestNG, is that it's annotated, so you have the
>>>>>> freedom there.
>>>>>>
>>>>>>
>>>>>>> If the test is added due to a regression, then it is put into
the
>>>> right
>>>>
>>>>>>> place in the test suite, and flagged with a comment (i.e. a
>>>> reference to
>>>>
>>>>>>> the Harmony JIRA number).
>>>>>> Yes - and I'd even advocate a parallel directory there too so that
>>>> it's
>>>>
>>>>>> clear that the regressions are different, but whatever.  The only
>>>> snag
>>>>
>>>>>> there is name collision with the classes.
>>>>> I thought we'd agreed that 'regression' was not a useful 
>> classification
>>>>> within the test suite layout ...
>>>>>
>>>>>
>>>>>> I think that a simple comment is enough.  If we want to get cute,
>>>> maybe
>>>>
>>>>>> a javadoc tag so we can manage mechanically in the future.
>>>>> ok -- do you have a usecase in mind?
>>>>>
>>>>> Regards,
>>>>> Tim
>>>>>
>>>>>
>>>>>
>>>>>>> George Harley1 wrote:
>>>>>>>
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> I think that regression tests should be marked in some
way.
>>>>>>>>
>>>>>>>> Agreed.  But can we please *resist* the temptation to do
this by
>>>>>>>> incorporating JIRA issue numbers into test case names (e.g.

> calling
>>>>>>>> unit test methods test_26() or test_JIRA_26()). I've seen
this 
> kind
>>>>>>>> of approach adopted in a couple of projects and, in my experience,
>>>> it
>>>>
>>>>>>>> often leads to the scattering of duplicated test code around
the
>>>> test
>>>>
>>>>>>>> harness.
>>>>>>>>
>>>>>>>> Better, methinks, to either create a new test method with
a
>>>>>>>> meaningful name or else augment an existing method - whatever

> makes
>>>>>>>> more sense for the particular issue. Then marking certain
code as
>>>>>>>> being for regression test purposes could be done in comments
that
>>>>>>>> include the URL of the JIRA issue. Perhaps an agreed tag
like
>>>> "JIRA"
>>>>
>>>>>>>> or "BUG" etc could be used as an eye-catcher as well ?
>>>>>>>> e.g.
>>>>>>>> // BUG http://issues.apache.org/jira/browse/HARMONY-26
>>>>>>>>
>>>>>>>>
>>>>>>>> My 2 Euro Cents.
>>>>>>>> Best regards, George
>>>>>>>> ________________________________________
>>>>>>>> George C. Harley
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> "Mishura, Stepan M" <stepan.m.mishura@intel.com> 12/01/2006
04:56
>>>>>>>> Please respond to
>>>>>>>> harmony-dev@incubator.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>>> To
>>>>>>>> <harmony-dev@incubator.apache.org>
>>>>>>>> cc
>>>>>>>>
>>>>>>>> Subject
>>>>>>>> RE: regression test suite
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Hello,
>>>>>>>> Tim Ellison wrote:
>>>>>>>> [snip]
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> What is the useful distinction for regression tests being
kept
>>>>>>>> separate?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> I can see that you may distinguish unit and 'system-level'
tests
>>>> just
>>>>
>>>>>>>>> because of the difference in frameworks etc. required,
but why do
>>>> I
>>>>
>>>>>>>> care
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> if the test was written due to a JIRA issue or test-based
>>>> development
>>>>
>>>>>>>> or
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> someone who get's kicks out of writing tests to break
the code?
>>>>>>>>>
>>>>>>>> I agree that separating regression tests doesn't make sense.
>>>>>>>> However I think that regression tests should be marked in
some 
> way.
>>>>>>>> This will signal a developer that a test was created to track
>>>> already
>>>>
>>>>>>>> known issue. IMHO, a regression test should point out to
a bug
>>>> report
>>>>
>>>>>>>> and a bug report (after resolving a bug) should contain a

> reference
>>>> to
>>>>
>>>>>>>> corresponding regression test in repository.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Stepan Mishura
>>>>>>>> Intel Middleware Products Division
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>> --
>>>>>
>>>>> Tim Ellison (t.p.ellison@gmail.com)
>>>>> IBM Java technology centre, UK.
>>> --
>>>
>>> Tim Ellison (t.p.ellison@gmail.com)
>>> IBM Java technology centre, UK.
>>>
>>
>>
> 
> 
> 

Mime
View raw message