harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Geir Magnusson Jr <g...@pobox.com>
Subject [classlib] Testing
Date Wed, 22 Mar 2006 12:15:28 GMT
Pulling out of the various threads where we have been discussing, can we 
agree on the problem :

We have unique problems compared to other Java projects because we need 
to find a way to reliably test the things that are commonly expected to 
be a solid point of reference - namely the core class library.

Further, we've been implicitly doing "integration testing" because - so 
far - the only way we've been testing our code has been 'in situ' in the 
VM - not in an isolated test harness.  To me, this turns it into an 
integration test.

Sure, we're using JUnit, but because of the fact we are implmenting core 
java.* APIs, we aren't testing with a framework that has been 
independently tested for correctness, like we would when testing any 
other code.

I hope I got that idea across - I believe that we have to go beyond 
normal testing approaches because we don't have a normal situation.

So I think there are three things we want to do (adopting the 
terminology that came from the discussion with Tim and Leo ) :

1) implementation tests
2) spec/API tests (I'll bundle together)
3) integration/functional tests

I believe that for #1, the issues related to being on the bootclasspath 
don't matter, because we aren't testing that aspect of the classes 
(which is how they behave integrated w/ the VM and security system) but 
rather the basic internal functioning.

I'm not sure how to approach this, but I'll try.  I'd love to hear how 
Sun, IBM or BEA deals with this, or be told why it isn't an issue :)

Implementation tests : I'd like to see us be able to do #1 via the 
standard same-package technique (i.e. testing a.b.C w/ a.b.CTest) but 
we'll run into a tangle of classloader problems, I suspect, becuase we 
want to be testing java.* code in a system that already has java.* code. 
  Can anyone see a way we can do this - test the classlibrary from the 
integration point of view - using some test harness + any known-good 
JRE, like Sun's or IBM's?

Spec/API tests : these are, IMO, a kind of integration test, because 
proper spec/API behavior *is* dependent on factors beyond the actual 
code itself (like classloader configuration, and security context). 
Because of this, the *.test.* pattern makes perfect sense.  Assuming we 
could produce something useful for #1 (i.e. a test harness/framework), 
could we then augment it to simulate the classloader config + security 
config that we'd get in a real VM?  That will give us the ability to 
test in isolation of the VM, and also let us 'break' the environment to 
ensure that the code fails in a predictable way.

Intgration/functional : this is a whole range of things, from doing the 
Spec/API tests in an actual VM, to the tests that exercise the code 
through interaction with external systems (like network, RMI, GUI, etc)

***

Now, it might be suggested that we just ignore the implementation 
testing (#1) and just do #2 and #3 as we are now, and hope we have a 
good enough test suite.  It could be argued that when Sun started, they 
didn't have a known-good platform to do implementation testing on like 
we do now.  I don't know if that's true.

The difference is that we need to produce something of the same quality 
as Sun's Java 5, not Sun's Java 1.0.  We've had 11 years since 1.0 to 
learn about testing, but they've had 11 years to get things solid.

What to do....

geir

Mime
View raw message