harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paulex Yang <paulex.y...@gmail.com>
Subject Re: [classlib] Testing
Date Wed, 26 Apr 2006 10:01:04 GMT
Richard Liang wrote:
> Mikhail Loenko wrote:
>> how about 'specific'? impl seems to be not very informative.
>>
>>   
> +1 to Oliver's impl :-)
>> I have a concern abou proposed package naming guidelines:
>> package name
>>     org.apache.harmony.security.tests.org.apache.harmony.security
>> is not much better then 1000-character long test name.
>>
>>   
> +1. I think the prefix org.apache.harmony.security is unnecessary. 
> "tests.impl.org.apache.harmony.security" is enough to tell people what 
> the test cases belong to
I agree it is duplicated and unnecessary, but it is a general convention 
for Java application to start with organization name, i.e., 
"org.apache.harmony". If I understand correctly, our implementation 
package's name must start with "org.apache.harmony.<module name>", so 
how about this:

org.apache.harmony.<module name>.tests.impl.<rest of package name>

as a example, test for org.apache.harmony.security.util will be named as:
org.apache.harmony.security.tests.impl.util

while the API test still be named as
org.apache.harmony.security.tests.api.java.security

comments?
>
> Any comments? Thanks a lot.
>> Thanks,
>> Mikhail
>>
>>
>> 2006/4/26, Paulex Yang <paulex.yang@gmail.com>:
>>  
>>> Oliver Deakin wrote:
>>>    
>>>> George Harley wrote:
>>>>      
>>>>> Mikhail Loenko wrote:
>>>>>        
>>>>>> Hello
>>>>>>
>>>>>> I'd like to bring this thread back.
>>>>>>
>>>>>> Number of tests is growing and it is time to put them in order.
>>>>>>
>>>>>> So far we may have:
>>>>>>
>>>>>> 1) implementation-specific tests that designed to be run from
>>>>>> bootclasspath
>>>>>> 2) implementation-specific tests that might be run from classpath
>>>>>> 3) implementation-specific tests that designed to be run from 
>>>>>> classpath
>>>>>> 4) implementation-independent tests that designed to be run from
>>>>>> bootclasspath
>>>>>> 5) implementation-independent tests that might be run from classpath
>>>>>> 6) implementation-independent tests that designed to be run from
>>>>>> classpath
>>>>>>
>>>>>> Also we seem to have the following packages, where the tests are:
>>>>>>
>>>>>> 1) the same package as implementation
>>>>>> 2) org.apache.harmony.tests.[the same package as implementation]
>>>>>> 3) tests.api.[the same package as implementation]
>>>>>>
>>>>>> I suggest that we work out step-by-step solution as we could not

>>>>>> reach
>>>>>> an agreement for the general universal one
>>>>>>
>>>>>> So for the first step I suggest that we separate i-independent tests
>>>>>> that must or may be run from classpath
>>>>>>
>>>>>> I suggest that we put them into package
>>>>>> tests.module.compatible.[package of implementation being tested]
>>>>>>
>>>>>> Comments?
>>>>>>
>>>>>> Thanks,
>>>>>> Mikhail
>>>>>>
>>>>>>           
>>>>> Hi Mikhail,
>>>>>
>>>>> I've just started working through the modules to merge test packages
>>>>> "tests.api.[same package as implementation]" and "tests.api.[same
>>>>> package as implementation]" into one package space. Using the class
>>>>> library package naming guidelines from off the web site [1], all of
>>>>> the tests for the text module have been consolidated under
>>>>> org.apache.harmony.text.tests.[package under test].
>>>>>
>>>>> Of course, the text module has only "implementation-independent tests
>>>>> that designed to be run from classpath". For modules that have got
>>>>> implementation-specific tests then I suppose we could use something
>>>>> like "org.apache.harmony.[module].tests.impl.[package under test]" or
>>>>> "org.apache.harmony.[module].tests.internal.[package under test]"
>>>>> etc. I've got no preference.
>>>>>         
>>>> I think impl is preferable over internal here, as we already use
>>>> internal in our implementation package names to indicate classes
>>>> totally internal to that bundle. To also use internal to label tests
>>>> that are implementation specific may cause confusion.
>>>>
>>>>       
>>> +1 from me.
>>>    
>>>> Regards,
>>>> Oliver
>>>>
>>>>      
>>>>> Best regards,
>>>>> George
>>>>>
>>>>>
>>>>> [1]
>>>>> http://incubator.apache.org/harmony/subcomponents/classlibrary/pkgnaming.html

>>>>>
>>>>>
>>>>>
>>>>>
>>>>>        
>>>>>> 2006/3/24, George Harley <george.c.harley@googlemail.com>:
>>>>>>
>>>>>>          
>>>>>>> Geir Magnusson Jr wrote:
>>>>>>>
>>>>>>>            
>>>>>>>> Leo Simons wrote:
>>>>>>>>
>>>>>>>>              
>>>>>>>>> On Wed, Mar 22, 2006 at 08:02:44AM -0500, Geir Magnusson
Jr 
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>                
>>>>>>>>>> Leo Simons wrote:
>>>>>>>>>>
>>>>>>>>>>                  
>>>>>>>>>>> On Wed, Mar 22, 2006 at 07:15:28AM -0500, Geir
Magnusson Jr 
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>                    
>>>>>>>>>>>> Pulling out of the various threads where
we have been 
>>>>>>>>>>>> discussing,
>>>>>>>>>>>> can we agree on the problem :
>>>>>>>>>>>>
>>>>>>>>>>>> We have unique problems compared to other
Java projects
>>>>>>>>>>>> because we
>>>>>>>>>>>> need to find a way to reliably test the things
that are 
>>>>>>>>>>>> commonly
>>>>>>>>>>>> expected to be a solid point of reference
- namely the core 
>>>>>>>>>>>> class
>>>>>>>>>>>> library.
>>>>>>>>>>>>
>>>>>>>>>>>> Further, we've been implicitly doing "integration
testing"
>>>>>>>>>>>> because
>>>>>>>>>>>> - so far - the only way we've been testing
our code has 
>>>>>>>>>>>> been 'in
>>>>>>>>>>>> situ' in the VM - not in an isolated test
harness.  To me, 
>>>>>>>>>>>> this
>>>>>>>>>>>> turns it into an integration test.
>>>>>>>>>>>>
>>>>>>>>>>>> Sure, we're using JUnit, but because of the
fact we are
>>>>>>>>>>>> implmenting core java.* APIs, we aren't testing
with a 
>>>>>>>>>>>> framework
>>>>>>>>>>>> that has been independently tested for correctness,
like we 
>>>>>>>>>>>> would
>>>>>>>>>>>> when testing any other code.
>>>>>>>>>>>>
>>>>>>>>>>>> I hope I got that idea across - I believe
that we have to go
>>>>>>>>>>>> beyond normal testing approaches because
we don't have a 
>>>>>>>>>>>> normal
>>>>>>>>>>>> situation.
>>>>>>>>>>>>
>>>>>>>>>>>>                       
>>>>>>>>>>> Where we define 'normal situation' as "running
a test 
>>>>>>>>>>> framework on
>>>>>>>>>>> top of
>>>>>>>>>>> the sun jdk and expecting any bugs to not be
in that jdk". 
>>>>>>>>>>> There's
>>>>>>>>>>> plenty
>>>>>>>>>>> of projects out there that have to test things
without having
>>>>>>>>>>> such a
>>>>>>>>>>> "stable reference JDK" luxury.....I imagine that
testing GCC is
>>>>>>>>>>> just as
>>>>>>>>>>> hard as this problem we have here :-)
>>>>>>>>>>>
>>>>>>>>>>>                     
>>>>>>>>>> Is it the same?  We need to have a running JVM+classlibarary
to
>>>>>>>>>> test
>>>>>>>>>> the classlibrary code.
>>>>>>>>>>
>>>>>>>>>>                   
>>>>>>>>> Well you need a working C compiler and standard C library
to
>>>>>>>>> compile the
>>>>>>>>> compiler so you can compile make so you can build bash
so you can
>>>>>>>>> run
>>>>>>>>> perl (which uses the standard C library functions all
over the
>>>>>>>>> place of
>>>>>>>>> course) so you can run the standard C library tests so
that 
>>>>>>>>> you know
>>>>>>>>> that
>>>>>>>>> the library you used when compiling the compiler were
correct so
>>>>>>>>> you can
>>>>>>>>> run the compiler tests. I don't think they actually do
things 
>>>>>>>>> that
>>>>>>>>> way, but
>>>>>>>>> it seems like basically the same problem. Having a virtual
>>>>>>>>> machine just
>>>>>>>>> makes it easier since you still assume "the native world"
as a
>>>>>>>>> baseline,
>>>>>>>>> which is a lot more than "the hardware".
>>>>>>>>>
>>>>>>>>>                 
>>>>>>>> There's a difference.  You can use a completely separate

>>>>>>>> toolchain to
>>>>>>>> build, test and verify the output of the C compiler.
>>>>>>>>
>>>>>>>> In our case, we are using the thing we are testing to test
itself.
>>>>>>>> There is no "known good" element possible right now.
>>>>>>>>
>>>>>>>> We use the classlibrary we are trying to test to execute
the test
>>>>>>>> framework that tests the classlibrary that is running it.
>>>>>>>>
>>>>>>>> The tool is testing itself.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>              
>>>>>>>>>>>> So I think there are three things we want
to do (adopting the
>>>>>>>>>>>> terminology that came from the discussion
with Tim and Leo ) :
>>>>>>>>>>>>
>>>>>>>>>>>> 1) implementation tests
>>>>>>>>>>>> 2) spec/API tests (I'll bundle together)
>>>>>>>>>>>> 3) integration/functional tests
>>>>>>>>>>>>
>>>>>>>>>>>> I believe that for #1, the issues related
to being on the
>>>>>>>>>>>> bootclasspath don't matter, because we aren't
testing that 
>>>>>>>>>>>> aspect
>>>>>>>>>>>> of the classes (which is how they behave
integrated w/ the 
>>>>>>>>>>>> VM and
>>>>>>>>>>>> security system) but rather the basic internal
functioning.
>>>>>>>>>>>>
>>>>>>>>>>>> I'm not sure how to approach this, but I'll
try.  I'd love to
>>>>>>>>>>>> hear
>>>>>>>>>>>> how Sun, IBM or BEA deals with this, or be
told why it 
>>>>>>>>>>>> isn't an
>>>>>>>>>>>> issue :)
>>>>>>>>>>>>
>>>>>>>>>>>> Implementation tests : I'd like to see us
be able to do #1 via
>>>>>>>>>>>> the
>>>>>>>>>>>> standard same-package technique (i.e. testing
a.b.C w/ 
>>>>>>>>>>>> a.b.CTest)
>>>>>>>>>>>> but we'll run into a tangle of classloader
problems, I 
>>>>>>>>>>>> suspect,
>>>>>>>>>>>> becuase we want to be testing java.* code
in a system that
>>>>>>>>>>>> already
>>>>>>>>>>>> has java.* code. Can anyone see a way we
can do this - test 
>>>>>>>>>>>> the
>>>>>>>>>>>> classlibrary from the integration point of
view - using 
>>>>>>>>>>>> some test
>>>>>>>>>>>> harness + any known-good JRE, like Sun's
or IBM's?
>>>>>>>>>>>>
>>>>>>>>>>>>                       
>>>>>>>>>>> Ew, that won't work in the end since we should
assume our 
>>>>>>>>>>> own JRE
>>>>>>>>>>> is going
>>>>>>>>>>> to be "known-better" :-). But it might be a nice
way to
>>>>>>>>>>> "bootstrap"
>>>>>>>>>>> (eg
>>>>>>>>>>> we test with an external JRE until we satisfy
the tests and
>>>>>>>>>>> then we
>>>>>>>>>>> switch
>>>>>>>>>>> to testing with an earlier build).
>>>>>>>>>>>
>>>>>>>>>>>                     
>>>>>>>>>> Lets be clear - even using our own "earlier build"
doesn't solve
>>>>>>>>>> the
>>>>>>>>>> problem I'm describing, because as it stands now,
we don't use
>>>>>>>>>> "earlier build" classes to test with - we use the
code we 
>>>>>>>>>> want to
>>>>>>>>>> test as the clsaslibrary for the JRE that's running
the test
>>>>>>>>>> framework.
>>>>>>>>>>
>>>>>>>>>> The classes that we are testing are also the classes
used by the
>>>>>>>>>> testing framework.  IOW, any of the java.* classes
that JUnit
>>>>>>>>>> itself
>>>>>>>>>> needs (ex. java.util.HashMap) are exactly the same

>>>>>>>>>> implementation
>>>>>>>>>> that it's testing.
>>>>>>>>>>
>>>>>>>>>> That's why I think it's subtly different than a "bootstrap

>>>>>>>>>> and use
>>>>>>>>>> version - 1 to test" problem.  See what I mean?
>>>>>>>>>>
>>>>>>>>>>                   
>>>>>>>>> Yeah yeah, I was already way beyond thinking "just" JUnit
is 
>>>>>>>>> usable
>>>>>>>>> for the
>>>>>>>>> kind of test you're describing. At some point, fundamentally,
you
>>>>>>>>> either trust
>>>>>>>>> something external (whether its the sun jdk or the intel
compiler
>>>>>>>>> designers,
>>>>>>>>> at some point you do draw a line) or you find a way to
bootstrap.
>>>>>>>>>
>>>>>>>>>                 
>>>>>>>> Well, we do trust the Sun JDK.
>>>>>>>>
>>>>>>>>
>>>>>>>>              
>>>>>>>>>> I'm very open to the idea that I'm missing something
here, 
>>>>>>>>>> but I'd
>>>>>>>>>> like to know that you see the issue - that when we
test, we have
>>>>>>>>>>
>>>>>>>>>>   VM + "classlib to be tested" + JUnit + testcases
>>>>>>>>>>
>>>>>>>>>> where the testcases are testing the classlib the
VM is running
>>>>>>>>>> JUnit
>>>>>>>>>> with.
>>>>>>>>>>
>>>>>>>>>> There never is isolation of the code being tested
:
>>>>>>>>>>
>>>>>>>>>>   VM + "known good classlib" + Junit + testcases
>>>>>>>>>>
>>>>>>>>>> unless we have some framework where
>>>>>>>>>>
>>>>>>>>>>   VM + "known good classlib" + JUnit
>>>>>>>>>>       + framework("classlib to be tested")
>>>>>>>>>>            + testcases
>>>>>>>>>>
>>>>>>>>>> and it's that notion of "framework()" that I'm advocating
we
>>>>>>>>>> explore.
>>>>>>>>>>
>>>>>>>>>>                   
>>>>>>>>> I'm all for exploring it, I just fundamentally don't
buy into the
>>>>>>>>> "known
>>>>>>>>> good" bit. What happens when the 'classlib to be tested'
is 
>>>>>>>>> 'known
>>>>>>>>> better' than the 'known good' one? How do you define
"known"? How
>>>>>>>>> do you
>>>>>>>>> define "good"?
>>>>>>>>>
>>>>>>>>>                 
>>>>>>>> Known?  Passed some set of tests. So it could be the Sun
JDK 
>>>>>>>> for the
>>>>>>>> VM + "known good" part.
>>>>>>>>
>>>>>>>> I think you intuitively understand this.  When you find a
bug 
>>>>>>>> in code
>>>>>>>> you are testing, you first assume it's your code, not the

>>>>>>>> framework,
>>>>>>>> right?  In our case, our framework is actually the code we
are
>>>>>>>> testing, so we have a bit of a logical conundrum.
>>>>>>>>
>>>>>>>>
>>>>>>>>               
>>>>>>> Hi Geir,
>>>>>>>
>>>>>>> The number of Harmony public API classes that get loaded just
to
>>>>>>> run the
>>>>>>> JUnit harness is a little over 200. The majority of these are

>>>>>>> out of
>>>>>>> LUNI with a very low number coming from each of Security, NIO,

>>>>>>> Archive
>>>>>>> and Text.
>>>>>>>
>>>>>>> Sure there is a circular dependency between what we are building
>>>>>>> and the
>>>>>>> framework we are using to test it but it appears to touch on
only a
>>>>>>> relatively small part of Harmony....IMHO.
>>>>>>>
>>>>>>> Best regards,
>>>>>>> George
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>            
>>>>>>>>>>> Further ideas...
>>>>>>>>>>>
>>>>>>>>>>> -> look at how the native world does testing
>>>>>>>>>>>   (hint: it usually has #ifdefs, uses perl along
the way, and
>>>>>>>>>>> it is
>>>>>>>>>>>   certainly
>>>>>>>>>>>    "messy")
>>>>>>>>>>>   -> emulate that
>>>>>>>>>>>
>>>>>>>>>>> -> build a bigger, better specification test
>>>>>>>>>>>   -> and somehow "prove" it is "good enough"
>>>>>>>>>>>
>>>>>>>>>>> -> build a bigger, better integration test
>>>>>>>>>>>   -> and somehow "prove" it is "good enough"
>>>>>>>>>>>
>>>>>>>>>>> I'll admit my primary interest is the last one...
>>>>>>>>>>>
>>>>>>>>>>>                     
>>>>>>>>>> The problem I see with the last one is that the "parameter
>>>>>>>>>> space" is
>>>>>>>>>> *huge*.
>>>>>>>>>>
>>>>>>>>>>                   
>>>>>>>>> Yeah, that's one of the things that makes it interesting.
>>>>>>>>> Fortunately
>>>>>>>>> open source does have many monkeys...
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                
>>>>>>>>>> I believe that your preference for the last one comes
from the
>>>>>>>>>> Monte-Carlo style approach that Gump uses - hope
that your test
>>>>>>>>>> suite has enough variance that you "push" the thing
being tested
>>>>>>>>>> through enough of the parameter space that you can
be 
>>>>>>>>>> comfortable
>>>>>>>>>> you would have exposed the bugs.  Maybe.
>>>>>>>>>>
>>>>>>>>>>                   
>>>>>>>>> Ooh, now its becoming rather abstract...
>>>>>>>>>
>>>>>>>>> Well, perhaps, but more of the gump approache comes from
the idea
>>>>>>>>> that
>>>>>>>>> the parameter space itself is also at some point defined
in
>>>>>>>>> software,
>>>>>>>>> which may have bugs of its own. You circumvent that by
making
>>>>>>>>> humans the
>>>>>>>>> parameter space (don't start about how humans are buggy.
We don't
>>>>>>>>> want to
>>>>>>>>> get into existialism or faith systems when talking about
unit
>>>>>>>>> testing do
>>>>>>>>> we?). The thing that gump enables is "many monkey QA"
- a way for
>>>>>>>>> thousands
>>>>>>>>> of human beings to concurrently make shared assertions
about
>>>>>>>>> software
>>>>>>>>> without actually needing all that much human interaction.
>>>>>>>>>
>>>>>>>>> More concretely, if harmony can run all known java software,

>>>>>>>>> and run
>>>>>>>>> it to
>>>>>>>>> the asserted satisfaction of all its developers, you
can trust 
>>>>>>>>> that
>>>>>>>>> you have
>>>>>>>>> covered all the /relevant/ parts of the parameter space
you
>>>>>>>>> describe.
>>>>>>>>>
>>>>>>>>>                 
>>>>>>>> Yes.  And when you can run all knownn Java software, let
me 
>>>>>>>> know :)
>>>>>>>> That's my point about the parameter space being huge.  Even

>>>>>>>> when you
>>>>>>>> reduce the definition to "that of all known Java software",
you 
>>>>>>>> still
>>>>>>>> have a huge problem on your hands.
>>>>>>>>
>>>>>>>>
>>>>>>>>              
>>>>>>>>> You
>>>>>>>>> will never get that level of trust when the assertions
are 
>>>>>>>>> made by
>>>>>>>>> software
>>>>>>>>> rather than humans. This is how open source leads to
software
>>>>>>>>> quality.
>>>>>>>>>
>>>>>>>>> Quoting myself, 'gump is the most misunderstood piece
of 
>>>>>>>>> software,
>>>>>>>>> ever'.
>>>>>>>>>
>>>>>>>>> cheers,
>>>>>>>>>
>>>>>>>>> Leo
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                 
>>>>>> ---------------------------------------------------------------------

>>>>>>
>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>>> For additional commands, e-mail: 
>>>>>> harmony-dev-help@incubator.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>           
>>>>> ---------------------------------------------------------------------
>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>> For additional commands, e-mail: 
>>>>> harmony-dev-help@incubator.apache.org
>>>>>
>>>>>
>>>>>         
>>> -- 
>>> Paulex Yang
>>> China Software Development Lab
>>> IBM
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>
>>>
>>>     
>>
>> ---------------------------------------------------------------------
>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>
>>
>>   
>
>


-- 
Paulex Yang
China Software Development Lab
IBM



---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org


Mime
View raw message