George Harley1 wrote:
> Hi, 
> 
>> This is not the goal of unit tests. They do not test API, they test the 
> code,
>> including package-access members etc. That is why unit tests must reside
>> in the same package as the classes.
> 
> This is radical stuff. I have never heard the argument before that unit 
> tests *must* share the same package name as the type under test. One 
> important reason why many think it is a bad idea to take this approach is 
> that the resulting unit test code becomes too closely coupled with the 
> code under test.

Huh?  It's a common practice - it gives the unit test the ability to 
really test the implementation because it has package level priv.

> 
> Think about it ... 
> 
> If unit test code can access every package-access member and method in a 
> given type then the test code will eventually reach a point where it 
> contains a lot of brittle assumptions about internal implementation. 

It can, yes.  OTOH, it is testing internal private methods, so yes, 
there's going to be those assumptions built in.

> Consequently, when implementation changes occur, the corresponding test 
> code will break. Note : I'm not talking about API changes breaking the 
> test code here, I'm talking about changes to protected and package-access 
> stuff which - by definition - the developer has chosen to keep non-public. 

Right - and the unit tests test that code.  How else do you test it in a 
fine-grained way?  Only via public methods?


> So as the internals change the unit tests break. And further internal 
> changes make the unit tests break. And ultimately maintaining the tests 
> with close-coupling to the internals of the type under test becomes a bit 
> of a pain and their overall value in the test harness diminishes because 
> minor refactorings to the type-under-test now take so darned long to carry 
> out. 

But it gives you very fine-grained coverage, if you want that.

geir

> 
> 
> Best regards, 
> George
> ________________________________________
> George C. Harley
> 
> 
> 
> 
> 
> Mikhail Loenko <mloenko@gmail.com> 
> 16/01/2006 11:58
> Please respond to
> harmony-dev@incubator.apache.org
> 
> 
> To
> harmony-dev@incubator.apache.org
> cc
> 
> Subject
> Re: Test suite layout
> 
> 
> 
> 
> 
> 
> On 1/16/06, Tim Ellison <t.p.ellison@gmail.com> wrote:
> ...
>> If your unit tests are intended to test API, then they should be calling
>> the API in the same manner that an application will call the API.  Just
> 
> This is not the goal of unit tests. They do not test API, they test the 
> code,
> including package-access members etc. That is why unit tests must reside
> in the same package as the classes.
> 
> Other kinds of tests like compatibility or functional tests call just
> public and protected methods so they can be in different packages.
> 
> Thanks,
> Mikhail Loenko
> Intel Middleware Products Division
> 
> 
> 
> 
> 
>> look at the number of places where we check to see if the classloader ==
>> null to see where this matters.
>>
>>> Some of them may depend on security manager component if
>>> you like to run a test in some security restricted environment. But it
>>> is not a big issue: the security manager component can be configured
>>> dynamically and you should install your custom security manager before
>>> running a test.
>> Yes, we should do that on the application classloader to test security
>> manager functionality.
>>
>> Regards,
>> Tim
>>
>>> What do you think?
>>>
>>> Thanks,
>>> Stepan Mishura
>>> Intel Middleware Products Division
>>>
>>>
>>>> -----Original Message-----
>>>> From: Tim Ellison [mailto:t.p.ellison@gmail.com]
>>>> Sent: Thursday, January 12, 2006 8:04 PM
>>>> To: harmony-dev@incubator.apache.org
>>>> Subject: Re: Test suite layout
>>>>
>>>> Geir Magnusson Jr wrote:
>>>>
>>>>> Tim Ellison wrote:
>>>> <snip>
>>>>
>>>>>> We would have written it as java.io.tests, but the java.<whatever>
>>>>>> namespace is reserved, so the formula is simply
>>>>>>
>>>>>> <package>.<type>  ->   org.apache.harmony.tests.<package>.<type>Test
>>>>>
>>>>> Ug - then you have the problem of not being in the namespace as what
>>> you
>>>
>>>>> are testing.
>>>>>
>>>>> THat's why people use parallel trees - so your test code is
>>> physically
>>>
>>>>> separate but you have freedom of package access.
>>>> For 'normal' application code then you can do this, but since we are
>>>> writing the java packages themselves then you come unstuck because the
>>>> java packages have to run on the bootclasspath, and the tests on the
>>>> application classpath.
>>>>
>>>> You don't want to run all your tests on the bootclasspath (because 
> then
>>>> it would not be subject to the same security sandboxing as 
> applications
>>>> using the APIs you are testing); and you cannot put java.<whatever> on
>>>> the application classpath because the VM will catch you out (you'd get
>>> a
>>>
>>>> linkage error IIRC).
>>>>
>>>>
>>>>>> This makes it clear what is being tested, and where to add new tests
>>>> etc.
>>>>
>>>>> So would
>>>>>
>>>>>  test/org/apache/harmony/io/TestFoo.java
>>>>>
>>>>> (to test something in org.apache.harmony.io, and arguable to test the
>>>>> Foo.java class in there.  (or TestFoo.java - it's early - no coffee
>>> yet)
>>>
>>>> Not sure what you are saying here... For java.<whatever> packages we
>>>> need a prefix on the test packages to keep the VM happy, but for
>>>> org.apache.harmony packages we can have either pre- or post-.
>>>>
>>>> I'd actually prefer a postfix of .tests for non-API packages, though I
>>>> can understand if people object to the inconsistency; so
>>>>
>>>> org.apache.harmony.tests.java.io.FileTest.java      <- test API
>>>> org.apache.harmony.io.tests.FileImpltest.java  <- test public methods
>>>>                                                 in our IO impl'
>>>>
>>>>
>>>>> Simiarly
>>>>>
>>>>>  test/java/util/TestMap.java
>>>>>
>>>>>
>>>>>> Then within the test class itself the methods are named after the
>>> method
>>>
>>>>>> under test, with a familar JNI-style encoding,  so we have things
>>> like:
>>>
>>>>>> org.apache.harmony.tests.java.io.FileTest contains
>>>>>>    public void test_ConstructorLjava_io_FileLjava_lang_String() {
>>>>>>    ...
>>>>>>    }
>>>>>>
>>>>>> and
>>>>>>
>>>>>> org.apache.harmony.tests.java.lang.FloatTest contains
>>>>>>    public void test_compareToLjava_lang_Float() {
>>>>>>    ...
>>>>>>    }
>>>>>
>>>>> ...or whatever the convention is for JUnit.  I think that's one of
>>> the
>>>
>>>>> nice things about TestNG, is that it's annotated, so you have the
>>>>> freedom there.
>>>>>
>>>>>
>>>>>> If the test is added due to a regression, then it is put into the
>>> right
>>>
>>>>>> place in the test suite, and flagged with a comment (i.e. a
>>> reference to
>>>
>>>>>> the Harmony JIRA number).
>>>>>
>>>>> Yes - and I'd even advocate a parallel directory there too so that
>>> it's
>>>
>>>>> clear that the regressions are different, but whatever.  The only
>>> snag
>>>
>>>>> there is name collision with the classes.
>>>> I thought we'd agreed that 'regression' was not a useful 
> classification
>>>> within the test suite layout ...
>>>>
>>>>
>>>>> I think that a simple comment is enough.  If we want to get cute,
>>> maybe
>>>
>>>>> a javadoc tag so we can manage mechanically in the future.
>>>> ok -- do you have a usecase in mind?
>>>>
>>>> Regards,
>>>> Tim
>>>>
>>>>
>>>>
>>>>>> George Harley1 wrote:
>>>>>>
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> I think that regression tests should be marked in some way.
>>>>>>>
>>>>>>>
>>>>>>> Agreed.  But can we please *resist* the temptation to do this by
>>>>>>> incorporating JIRA issue numbers into test case names (e.g. calling
>>>>>>> unit test methods test_26() or test_JIRA_26()). I've seen this kind
>>>>>>> of approach adopted in a couple of projects and, in my experience,
>>> it
>>>
>>>>>>> often leads to the scattering of duplicated test code around the
>>> test
>>>
>>>>>>> harness.
>>>>>>>
>>>>>>> Better, methinks, to either create a new test method with a
>>>>>>> meaningful name or else augment an existing method - whatever makes
>>>>>>> more sense for the particular issue. Then marking certain code as
>>>>>>> being for regression test purposes could be done in comments that
>>>>>>> include the URL of the JIRA issue. Perhaps an agreed tag like
>>> "JIRA"
>>>
>>>>>>> or "BUG" etc could be used as an eye-catcher as well ?
>>>>>>> e.g.
>>>>>>> // BUG http://issues.apache.org/jira/browse/HARMONY-26
>>>>>>>
>>>>>>>
>>>>>>> My 2 Euro Cents.
>>>>>>> Best regards, George
>>>>>>> ________________________________________
>>>>>>> George C. Harley
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> "Mishura, Stepan M" <stepan.m.mishura@intel.com> 12/01/2006 04:56
>>>>>>> Please respond to
>>>>>>> harmony-dev@incubator.apache.org
>>>>>>>
>>>>>>>
>>>>>>> To
>>>>>>> <harmony-dev@incubator.apache.org>
>>>>>>> cc
>>>>>>>
>>>>>>> Subject
>>>>>>> RE: regression test suite
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Hello,
>>>>>>> Tim Ellison wrote:
>>>>>>> [snip]
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> What is the useful distinction for regression tests being kept
>>>>>>>
>>>>>>> separate?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> I can see that you may distinguish unit and 'system-level' tests
>>> just
>>>
>>>>>>>> because of the difference in frameworks etc. required, but why do
>>> I
>>>
>>>>>>> care
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> if the test was written due to a JIRA issue or test-based
>>> development
>>>
>>>>>>> or
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> someone who get's kicks out of writing tests to break the code?
>>>>>>>>
>>>>>>>
>>>>>>> I agree that separating regression tests doesn't make sense.
>>>>>>> However I think that regression tests should be marked in some way.
>>>>>>> This will signal a developer that a test was created to track
>>> already
>>>
>>>>>>> known issue. IMHO, a regression test should point out to a bug
>>> report
>>>
>>>>>>> and a bug report (after resolving a bug) should contain a reference
>>> to
>>>
>>>>>>> corresponding regression test in repository.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Stepan Mishura
>>>>>>> Intel Middleware Products Division
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>> --
>>>>
>>>> Tim Ellison (t.p.ellison@gmail.com)
>>>> IBM Java technology centre, UK.
>>>
>> --
>>
>> Tim Ellison (t.p.ellison@gmail.com)
>> IBM Java technology centre, UK.
>>
> 
> 
>