harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mishura, Stepan M" <stepan.m.mish...@intel.com>
Subject RE: Test suite layout
Date Fri, 13 Jan 2006 14:15:24 GMT
Tim Ellison wrote:

>
>For 'normal' application code then you can do this, but since we are
>writing the java packages themselves then you come unstuck because the
>java packages have to run on the bootclasspath, and the tests on the
>application classpath.
>
>You don't want to run all your tests on the bootclasspath (because then
>it would not be subject to the same security sandboxing as applications
>using the APIs you are testing); and you cannot put java.<whatever> on
>the application classpath because the VM will catch you out (you'd get
a
>linkage error IIRC).
>

Not 'all' tests - just unit tests for classlib :-)

I don't see big problems with sandbox and running unit tests on the
bootclasspath. The sandbox consists of three elements: verifier, class
loader and security manager. IMHO, most of classlib unit tests don't
depend on sandbox and are designed to verify tested class functionality
only. For example, if I want to verify that 1+1=2 why I have to remember
about sandbox? Some of them may depend on security manager component if
you like to run a test in some security restricted environment. But it
is not a big issue: the security manager component can be configured
dynamically and you should install your custom security manager before
running a test.

What do you think?

Thanks,
Stepan Mishura
Intel Middleware Products Division

>-----Original Message-----
>From: Tim Ellison [mailto:t.p.ellison@gmail.com]
>Sent: Thursday, January 12, 2006 8:04 PM
>To: harmony-dev@incubator.apache.org
>Subject: Re: Test suite layout
>
>Geir Magnusson Jr wrote:
>> Tim Ellison wrote:
><snip>
>
>>> We would have written it as java.io.tests, but the java.<whatever>
>>> namespace is reserved, so the formula is simply
>>>
>>> <package>.<type>  ->   org.apache.harmony.tests.<package>.<type>Test
>>
>>
>> Ug - then you have the problem of not being in the namespace as what
you
>> are testing.
>>
>> THat's why people use parallel trees - so your test code is
physically
>> separate but you have freedom of package access.
>
>For 'normal' application code then you can do this, but since we are
>writing the java packages themselves then you come unstuck because the
>java packages have to run on the bootclasspath, and the tests on the
>application classpath.
>
>You don't want to run all your tests on the bootclasspath (because then
>it would not be subject to the same security sandboxing as applications
>using the APIs you are testing); and you cannot put java.<whatever> on
>the application classpath because the VM will catch you out (you'd get
a
>linkage error IIRC).
>
>>> This makes it clear what is being tested, and where to add new tests
>etc.
>>
>>
>> So would
>>
>>   test/org/apache/harmony/io/TestFoo.java
>>
>> (to test something in org.apache.harmony.io, and arguable to test the
>> Foo.java class in there.  (or TestFoo.java - it's early - no coffee
yet)
>
>Not sure what you are saying here... For java.<whatever> packages we
>need a prefix on the test packages to keep the VM happy, but for
>org.apache.harmony packages we can have either pre- or post-.
>
>I'd actually prefer a postfix of .tests for non-API packages, though I
>can understand if people object to the inconsistency; so
>
>org.apache.harmony.tests.java.io.FileTest.java      <- test API
>org.apache.harmony.io.tests.FileImpltest.java  <- test public methods
>                                                  in our IO impl'
>
>> Simiarly
>>
>>   test/java/util/TestMap.java
>>
>>>
>>> Then within the test class itself the methods are named after the
method
>>> under test, with a familar JNI-style encoding,  so we have things
like:
>>>
>>> org.apache.harmony.tests.java.io.FileTest contains
>>>     public void test_ConstructorLjava_io_FileLjava_lang_String() {
>>>     ...
>>>     }
>>>
>>> and
>>>
>>> org.apache.harmony.tests.java.lang.FloatTest contains
>>>     public void test_compareToLjava_lang_Float() {
>>>     ...
>>>     }
>>
>>
>> ...or whatever the convention is for JUnit.  I think that's one of
the
>> nice things about TestNG, is that it's annotated, so you have the
>> freedom there.
>>
>>>
>>>
>>> If the test is added due to a regression, then it is put into the
right
>>> place in the test suite, and flagged with a comment (i.e. a
reference to
>>> the Harmony JIRA number).
>>
>>
>> Yes - and I'd even advocate a parallel directory there too so that
it's
>> clear that the regressions are different, but whatever.  The only
snag
>> there is name collision with the classes.
>
>I thought we'd agreed that 'regression' was not a useful classification
>within the test suite layout ...
>
>> I think that a simple comment is enough.  If we want to get cute,
maybe
>> a javadoc tag so we can manage mechanically in the future.
>
>ok -- do you have a usecase in mind?
>
>Regards,
>Tim
>
>
>>> George Harley1 wrote:
>>>
>>>> Hi,
>>>>
>>>>
>>>>> I think that regression tests should be marked in some way.
>>>>
>>>>
>>>>
>>>> Agreed.  But can we please *resist* the temptation to do this by
>>>> incorporating JIRA issue numbers into test case names (e.g. calling
>>>> unit test methods test_26() or test_JIRA_26()). I've seen this kind
>>>> of approach adopted in a couple of projects and, in my experience,
it
>>>> often leads to the scattering of duplicated test code around the
test
>>>> harness.
>>>>
>>>> Better, methinks, to either create a new test method with a
>>>> meaningful name or else augment an existing method - whatever makes
>>>> more sense for the particular issue. Then marking certain code as
>>>> being for regression test purposes could be done in comments that
>>>> include the URL of the JIRA issue. Perhaps an agreed tag like
"JIRA"
>>>> or "BUG" etc could be used as an eye-catcher as well ?
>>>> e.g.
>>>> // BUG http://issues.apache.org/jira/browse/HARMONY-26
>>>>
>>>>
>>>> My 2 Euro Cents.
>>>> Best regards, George
>>>> ________________________________________
>>>> George C. Harley
>>>>
>>>>
>>>>
>>>>
>>>> "Mishura, Stepan M" <stepan.m.mishura@intel.com> 12/01/2006 04:56
>>>> Please respond to
>>>> harmony-dev@incubator.apache.org
>>>>
>>>>
>>>> To
>>>> <harmony-dev@incubator.apache.org>
>>>> cc
>>>>
>>>> Subject
>>>> RE: regression test suite
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Hello,
>>>> Tim Ellison wrote:
>>>> [snip]
>>>>
>>>>
>>>>> What is the useful distinction for regression tests being kept
>>>>
>>>>
>>>> separate?
>>>>
>>>>
>>>>> I can see that you may distinguish unit and 'system-level' tests
just
>>>>> because of the difference in frameworks etc. required, but why do
I
>>>>
>>>>
>>>> care
>>>>
>>>>
>>>>> if the test was written due to a JIRA issue or test-based
development
>>>>
>>>>
>>>> or
>>>>
>>>>
>>>>> someone who get's kicks out of writing tests to break the code?
>>>>>
>>>>
>>>>
>>>> I agree that separating regression tests doesn't make sense.
>>>> However I think that regression tests should be marked in some way.
>>>> This will signal a developer that a test was created to track
already
>>>> known issue. IMHO, a regression test should point out to a bug
report
>>>> and a bug report (after resolving a bug) should contain a reference
to
>>>> corresponding regression test in repository.
>>>>
>>>> Thanks,
>>>> Stepan Mishura
>>>> Intel Middleware Products Division
>>>>
>>>>
>>>>
>>>
>>>
>>
>
>--
>
>Tim Ellison (t.p.ellison@gmail.com)
>IBM Java technology centre, UK.

Mime
View raw message