harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mikhail Loenko" <mloe...@gmail.com>
Subject Re: [classlib] Testing
Date Wed, 26 Apr 2006 10:59:21 GMT
Seems like implemantation-specific tests for java.security package do not
fit well into the proposed solution

I think we do not have to stick to requirement about
organization name in the tests: we already have tests in package java.*
why don't have package tests.*?

Thanks,
Mikhail

2006/4/26, Paulex Yang <paulex.yang@gmail.com>:
> Richard Liang wrote:
> > Mikhail Loenko wrote:
> >> how about 'specific'? impl seems to be not very informative.
> >>
> >>
> > +1 to Oliver's impl :-)
> >> I have a concern abou proposed package naming guidelines:
> >> package name
> >>     org.apache.harmony.security.tests.org.apache.harmony.security
> >> is not much better then 1000-character long test name.
> >>
> >>
> > +1. I think the prefix org.apache.harmony.security is unnecessary.
> > "tests.impl.org.apache.harmony.security" is enough to tell people what
> > the test cases belong to
> I agree it is duplicated and unnecessary, but it is a general convention
> for Java application to start with organization name, i.e.,
> "org.apache.harmony". If I understand correctly, our implementation
> package's name must start with "org.apache.harmony.<module name>", so
> how about this:
>
> org.apache.harmony.<module name>.tests.impl.<rest of package name>
>
> as a example, test for org.apache.harmony.security.util will be named as:
> org.apache.harmony.security.tests.impl.util
>
> while the API test still be named as
> org.apache.harmony.security.tests.api.java.security
>
> comments?
> >
> > Any comments? Thanks a lot.
> >> Thanks,
> >> Mikhail
> >>
> >>
> >> 2006/4/26, Paulex Yang <paulex.yang@gmail.com>:
> >>
> >>> Oliver Deakin wrote:
> >>>
> >>>> George Harley wrote:
> >>>>
> >>>>> Mikhail Loenko wrote:
> >>>>>
> >>>>>> Hello
> >>>>>>
> >>>>>> I'd like to bring this thread back.
> >>>>>>
> >>>>>> Number of tests is growing and it is time to put them in order.
> >>>>>>
> >>>>>> So far we may have:
> >>>>>>
> >>>>>> 1) implementation-specific tests that designed to be run from
> >>>>>> bootclasspath
> >>>>>> 2) implementation-specific tests that might be run from classpath
> >>>>>> 3) implementation-specific tests that designed to be run from
> >>>>>> classpath
> >>>>>> 4) implementation-independent tests that designed to be run
from
> >>>>>> bootclasspath
> >>>>>> 5) implementation-independent tests that might be run from classpath
> >>>>>> 6) implementation-independent tests that designed to be run
from
> >>>>>> classpath
> >>>>>>
> >>>>>> Also we seem to have the following packages, where the tests
are:
> >>>>>>
> >>>>>> 1) the same package as implementation
> >>>>>> 2) org.apache.harmony.tests.[the same package as implementation]
> >>>>>> 3) tests.api.[the same package as implementation]
> >>>>>>
> >>>>>> I suggest that we work out step-by-step solution as we could
not
> >>>>>> reach
> >>>>>> an agreement for the general universal one
> >>>>>>
> >>>>>> So for the first step I suggest that we separate i-independent
tests
> >>>>>> that must or may be run from classpath
> >>>>>>
> >>>>>> I suggest that we put them into package
> >>>>>> tests.module.compatible.[package of implementation being tested]
> >>>>>>
> >>>>>> Comments?
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Mikhail
> >>>>>>
> >>>>>>
> >>>>> Hi Mikhail,
> >>>>>
> >>>>> I've just started working through the modules to merge test packages
> >>>>> "tests.api.[same package as implementation]" and "tests.api.[same
> >>>>> package as implementation]" into one package space. Using the class
> >>>>> library package naming guidelines from off the web site [1], all
of
> >>>>> the tests for the text module have been consolidated under
> >>>>> org.apache.harmony.text.tests.[package under test].
> >>>>>
> >>>>> Of course, the text module has only "implementation-independent
tests
> >>>>> that designed to be run from classpath". For modules that have got
> >>>>> implementation-specific tests then I suppose we could use something
> >>>>> like "org.apache.harmony.[module].tests.impl.[package under test]"
or
> >>>>> "org.apache.harmony.[module].tests.internal.[package under test]"
> >>>>> etc. I've got no preference.
> >>>>>
> >>>> I think impl is preferable over internal here, as we already use
> >>>> internal in our implementation package names to indicate classes
> >>>> totally internal to that bundle. To also use internal to label tests
> >>>> that are implementation specific may cause confusion.
> >>>>
> >>>>
> >>> +1 from me.
> >>>
> >>>> Regards,
> >>>> Oliver
> >>>>
> >>>>
> >>>>> Best regards,
> >>>>> George
> >>>>>
> >>>>>
> >>>>> [1]
> >>>>> http://incubator.apache.org/harmony/subcomponents/classlibrary/pkgnaming.html
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>> 2006/3/24, George Harley <george.c.harley@googlemail.com>:
> >>>>>>
> >>>>>>
> >>>>>>> Geir Magnusson Jr wrote:
> >>>>>>>
> >>>>>>>
> >>>>>>>> Leo Simons wrote:
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> On Wed, Mar 22, 2006 at 08:02:44AM -0500, Geir Magnusson
Jr
> >>>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> Leo Simons wrote:
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>> On Wed, Mar 22, 2006 at 07:15:28AM -0500,
Geir Magnusson Jr
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>> Pulling out of the various threads where
we have been
> >>>>>>>>>>>> discussing,
> >>>>>>>>>>>> can we agree on the problem :
> >>>>>>>>>>>>
> >>>>>>>>>>>> We have unique problems compared to
other Java projects
> >>>>>>>>>>>> because we
> >>>>>>>>>>>> need to find a way to reliably test
the things that are
> >>>>>>>>>>>> commonly
> >>>>>>>>>>>> expected to be a solid point of reference
- namely the core
> >>>>>>>>>>>> class
> >>>>>>>>>>>> library.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Further, we've been implicitly doing
"integration testing"
> >>>>>>>>>>>> because
> >>>>>>>>>>>> - so far - the only way we've been testing
our code has
> >>>>>>>>>>>> been 'in
> >>>>>>>>>>>> situ' in the VM - not in an isolated
test harness.  To me,
> >>>>>>>>>>>> this
> >>>>>>>>>>>> turns it into an integration test.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Sure, we're using JUnit, but because
of the fact we are
> >>>>>>>>>>>> implmenting core java.* APIs, we aren't
testing with a
> >>>>>>>>>>>> framework
> >>>>>>>>>>>> that has been independently tested for
correctness, like we
> >>>>>>>>>>>> would
> >>>>>>>>>>>> when testing any other code.
> >>>>>>>>>>>>
> >>>>>>>>>>>> I hope I got that idea across - I believe
that we have to go
> >>>>>>>>>>>> beyond normal testing approaches because
we don't have a
> >>>>>>>>>>>> normal
> >>>>>>>>>>>> situation.
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>> Where we define 'normal situation' as "running
a test
> >>>>>>>>>>> framework on
> >>>>>>>>>>> top of
> >>>>>>>>>>> the sun jdk and expecting any bugs to not
be in that jdk".
> >>>>>>>>>>> There's
> >>>>>>>>>>> plenty
> >>>>>>>>>>> of projects out there that have to test
things without having
> >>>>>>>>>>> such a
> >>>>>>>>>>> "stable reference JDK" luxury.....I imagine
that testing GCC is
> >>>>>>>>>>> just as
> >>>>>>>>>>> hard as this problem we have here :-)
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>> Is it the same?  We need to have a running JVM+classlibarary
to
> >>>>>>>>>> test
> >>>>>>>>>> the classlibrary code.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>> Well you need a working C compiler and standard
C library to
> >>>>>>>>> compile the
> >>>>>>>>> compiler so you can compile make so you can build
bash so you can
> >>>>>>>>> run
> >>>>>>>>> perl (which uses the standard C library functions
all over the
> >>>>>>>>> place of
> >>>>>>>>> course) so you can run the standard C library tests
so that
> >>>>>>>>> you know
> >>>>>>>>> that
> >>>>>>>>> the library you used when compiling the compiler
were correct so
> >>>>>>>>> you can
> >>>>>>>>> run the compiler tests. I don't think they actually
do things
> >>>>>>>>> that
> >>>>>>>>> way, but
> >>>>>>>>> it seems like basically the same problem. Having
a virtual
> >>>>>>>>> machine just
> >>>>>>>>> makes it easier since you still assume "the native
world" as a
> >>>>>>>>> baseline,
> >>>>>>>>> which is a lot more than "the hardware".
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>> There's a difference.  You can use a completely separate
> >>>>>>>> toolchain to
> >>>>>>>> build, test and verify the output of the C compiler.
> >>>>>>>>
> >>>>>>>> In our case, we are using the thing we are testing to
test itself.
> >>>>>>>> There is no "known good" element possible right now.
> >>>>>>>>
> >>>>>>>> We use the classlibrary we are trying to test to execute
the test
> >>>>>>>> framework that tests the classlibrary that is running
it.
> >>>>>>>>
> >>>>>>>> The tool is testing itself.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>>>>> So I think there are three things we
want to do (adopting the
> >>>>>>>>>>>> terminology that came from the discussion
with Tim and Leo ) :
> >>>>>>>>>>>>
> >>>>>>>>>>>> 1) implementation tests
> >>>>>>>>>>>> 2) spec/API tests (I'll bundle together)
> >>>>>>>>>>>> 3) integration/functional tests
> >>>>>>>>>>>>
> >>>>>>>>>>>> I believe that for #1, the issues related
to being on the
> >>>>>>>>>>>> bootclasspath don't matter, because
we aren't testing that
> >>>>>>>>>>>> aspect
> >>>>>>>>>>>> of the classes (which is how they behave
integrated w/ the
> >>>>>>>>>>>> VM and
> >>>>>>>>>>>> security system) but rather the basic
internal functioning.
> >>>>>>>>>>>>
> >>>>>>>>>>>> I'm not sure how to approach this, but
I'll try.  I'd love to
> >>>>>>>>>>>> hear
> >>>>>>>>>>>> how Sun, IBM or BEA deals with this,
or be told why it
> >>>>>>>>>>>> isn't an
> >>>>>>>>>>>> issue :)
> >>>>>>>>>>>>
> >>>>>>>>>>>> Implementation tests : I'd like to see
us be able to do #1 via
> >>>>>>>>>>>> the
> >>>>>>>>>>>> standard same-package technique (i.e.
testing a.b.C w/
> >>>>>>>>>>>> a.b.CTest)
> >>>>>>>>>>>> but we'll run into a tangle of classloader
problems, I
> >>>>>>>>>>>> suspect,
> >>>>>>>>>>>> becuase we want to be testing java.*
code in a system that
> >>>>>>>>>>>> already
> >>>>>>>>>>>> has java.* code. Can anyone see a way
we can do this - test
> >>>>>>>>>>>> the
> >>>>>>>>>>>> classlibrary from the integration point
of view - using
> >>>>>>>>>>>> some test
> >>>>>>>>>>>> harness + any known-good JRE, like Sun's
or IBM's?
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>> Ew, that won't work in the end since we
should assume our
> >>>>>>>>>>> own JRE
> >>>>>>>>>>> is going
> >>>>>>>>>>> to be "known-better" :-). But it might be
a nice way to
> >>>>>>>>>>> "bootstrap"
> >>>>>>>>>>> (eg
> >>>>>>>>>>> we test with an external JRE until we satisfy
the tests and
> >>>>>>>>>>> then we
> >>>>>>>>>>> switch
> >>>>>>>>>>> to testing with an earlier build).
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>> Lets be clear - even using our own "earlier
build" doesn't solve
> >>>>>>>>>> the
> >>>>>>>>>> problem I'm describing, because as it stands
now, we don't use
> >>>>>>>>>> "earlier build" classes to test with - we use
the code we
> >>>>>>>>>> want to
> >>>>>>>>>> test as the clsaslibrary for the JRE that's
running the test
> >>>>>>>>>> framework.
> >>>>>>>>>>
> >>>>>>>>>> The classes that we are testing are also the
classes used by the
> >>>>>>>>>> testing framework.  IOW, any of the java.* classes
that JUnit
> >>>>>>>>>> itself
> >>>>>>>>>> needs (ex. java.util.HashMap) are exactly the
same
> >>>>>>>>>> implementation
> >>>>>>>>>> that it's testing.
> >>>>>>>>>>
> >>>>>>>>>> That's why I think it's subtly different than
a "bootstrap
> >>>>>>>>>> and use
> >>>>>>>>>> version - 1 to test" problem.  See what I mean?
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>> Yeah yeah, I was already way beyond thinking "just"
JUnit is
> >>>>>>>>> usable
> >>>>>>>>> for the
> >>>>>>>>> kind of test you're describing. At some point, fundamentally,
you
> >>>>>>>>> either trust
> >>>>>>>>> something external (whether its the sun jdk or the
intel compiler
> >>>>>>>>> designers,
> >>>>>>>>> at some point you do draw a line) or you find a
way to bootstrap.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>> Well, we do trust the Sun JDK.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>>> I'm very open to the idea that I'm missing something
here,
> >>>>>>>>>> but I'd
> >>>>>>>>>> like to know that you see the issue - that when
we test, we have
> >>>>>>>>>>
> >>>>>>>>>>   VM + "classlib to be tested" + JUnit + testcases
> >>>>>>>>>>
> >>>>>>>>>> where the testcases are testing the classlib
the VM is running
> >>>>>>>>>> JUnit
> >>>>>>>>>> with.
> >>>>>>>>>>
> >>>>>>>>>> There never is isolation of the code being tested
:
> >>>>>>>>>>
> >>>>>>>>>>   VM + "known good classlib" + Junit + testcases
> >>>>>>>>>>
> >>>>>>>>>> unless we have some framework where
> >>>>>>>>>>
> >>>>>>>>>>   VM + "known good classlib" + JUnit
> >>>>>>>>>>       + framework("classlib to be tested")
> >>>>>>>>>>            + testcases
> >>>>>>>>>>
> >>>>>>>>>> and it's that notion of "framework()" that I'm
advocating we
> >>>>>>>>>> explore.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>> I'm all for exploring it, I just fundamentally don't
buy into the
> >>>>>>>>> "known
> >>>>>>>>> good" bit. What happens when the 'classlib to be
tested' is
> >>>>>>>>> 'known
> >>>>>>>>> better' than the 'known good' one? How do you define
"known"? How
> >>>>>>>>> do you
> >>>>>>>>> define "good"?
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>> Known?  Passed some set of tests. So it could be the
Sun JDK
> >>>>>>>> for the
> >>>>>>>> VM + "known good" part.
> >>>>>>>>
> >>>>>>>> I think you intuitively understand this.  When you find
a bug
> >>>>>>>> in code
> >>>>>>>> you are testing, you first assume it's your code, not
the
> >>>>>>>> framework,
> >>>>>>>> right?  In our case, our framework is actually the code
we are
> >>>>>>>> testing, so we have a bit of a logical conundrum.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>> Hi Geir,
> >>>>>>>
> >>>>>>> The number of Harmony public API classes that get loaded
just to
> >>>>>>> run the
> >>>>>>> JUnit harness is a little over 200. The majority of these
are
> >>>>>>> out of
> >>>>>>> LUNI with a very low number coming from each of Security,
NIO,
> >>>>>>> Archive
> >>>>>>> and Text.
> >>>>>>>
> >>>>>>> Sure there is a circular dependency between what we are
building
> >>>>>>> and the
> >>>>>>> framework we are using to test it but it appears to touch
on only a
> >>>>>>> relatively small part of Harmony....IMHO.
> >>>>>>>
> >>>>>>> Best regards,
> >>>>>>> George
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>> Further ideas...
> >>>>>>>>>>>
> >>>>>>>>>>> -> look at how the native world does
testing
> >>>>>>>>>>>   (hint: it usually has #ifdefs, uses perl
along the way, and
> >>>>>>>>>>> it is
> >>>>>>>>>>>   certainly
> >>>>>>>>>>>    "messy")
> >>>>>>>>>>>   -> emulate that
> >>>>>>>>>>>
> >>>>>>>>>>> -> build a bigger, better specification
test
> >>>>>>>>>>>   -> and somehow "prove" it is "good
enough"
> >>>>>>>>>>>
> >>>>>>>>>>> -> build a bigger, better integration
test
> >>>>>>>>>>>   -> and somehow "prove" it is "good
enough"
> >>>>>>>>>>>
> >>>>>>>>>>> I'll admit my primary interest is the last
one...
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>> The problem I see with the last one is that
the "parameter
> >>>>>>>>>> space" is
> >>>>>>>>>> *huge*.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>> Yeah, that's one of the things that makes it interesting.
> >>>>>>>>> Fortunately
> >>>>>>>>> open source does have many monkeys...
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> I believe that your preference for the last
one comes from the
> >>>>>>>>>> Monte-Carlo style approach that Gump uses -
hope that your test
> >>>>>>>>>> suite has enough variance that you "push" the
thing being tested
> >>>>>>>>>> through enough of the parameter space that you
can be
> >>>>>>>>>> comfortable
> >>>>>>>>>> you would have exposed the bugs.  Maybe.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>> Ooh, now its becoming rather abstract...
> >>>>>>>>>
> >>>>>>>>> Well, perhaps, but more of the gump approache comes
from the idea
> >>>>>>>>> that
> >>>>>>>>> the parameter space itself is also at some point
defined in
> >>>>>>>>> software,
> >>>>>>>>> which may have bugs of its own. You circumvent that
by making
> >>>>>>>>> humans the
> >>>>>>>>> parameter space (don't start about how humans are
buggy. We don't
> >>>>>>>>> want to
> >>>>>>>>> get into existialism or faith systems when talking
about unit
> >>>>>>>>> testing do
> >>>>>>>>> we?). The thing that gump enables is "many monkey
QA" - a way for
> >>>>>>>>> thousands
> >>>>>>>>> of human beings to concurrently make shared assertions
about
> >>>>>>>>> software
> >>>>>>>>> without actually needing all that much human interaction.
> >>>>>>>>>
> >>>>>>>>> More concretely, if harmony can run all known java
software,
> >>>>>>>>> and run
> >>>>>>>>> it to
> >>>>>>>>> the asserted satisfaction of all its developers,
you can trust
> >>>>>>>>> that
> >>>>>>>>> you have
> >>>>>>>>> covered all the /relevant/ parts of the parameter
space you
> >>>>>>>>> describe.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>> Yes.  And when you can run all knownn Java software,
let me
> >>>>>>>> know :)
> >>>>>>>> That's my point about the parameter space being huge.
 Even
> >>>>>>>> when you
> >>>>>>>> reduce the definition to "that of all known Java software",
you
> >>>>>>>> still
> >>>>>>>> have a huge problem on your hands.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> You
> >>>>>>>>> will never get that level of trust when the assertions
are
> >>>>>>>>> made by
> >>>>>>>>> software
> >>>>>>>>> rather than humans. This is how open source leads
to software
> >>>>>>>>> quality.
> >>>>>>>>>
> >>>>>>>>> Quoting myself, 'gump is the most misunderstood
piece of
> >>>>>>>>> software,
> >>>>>>>>> ever'.
> >>>>>>>>>
> >>>>>>>>> cheers,
> >>>>>>>>>
> >>>>>>>>> Leo
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>> ---------------------------------------------------------------------
> >>>>>>
> >>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >>>>>> For additional commands, e-mail:
> >>>>>> harmony-dev-help@incubator.apache.org
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>> ---------------------------------------------------------------------
> >>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >>>>> For additional commands, e-mail:
> >>>>> harmony-dev-help@incubator.apache.org
> >>>>>
> >>>>>
> >>>>>
> >>> --
> >>> Paulex Yang
> >>> China Software Development Lab
> >>> IBM
> >>>
> >>>
> >>>
> >>> ---------------------------------------------------------------------
> >>> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >>>
> >>>
> >>>
> >>
> >> ---------------------------------------------------------------------
> >> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >>
> >>
> >>
> >
> >
>
>
> --
> Paulex Yang
> China Software Development Lab
> IBM
>
>
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>

---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org


Mime
View raw message