harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stepan Mishura" <stepan.mish...@gmail.com>
Subject Re: classlib test suite status emails?
Date Wed, 12 Apr 2006 04:59:41 GMT
Hi George,

 Your example looks good for me and I think everybody agreed that we should
organize testing to avoid running all tests for each update: if you fix bug
in 'net' module you don't have to run tests say for 'awt' module but if you
update 'luni' then you have to run tests for all modules.

Thanks,
Stepan.


On 4/11/06, George Harley wrote:
>
> Stepan Mishura wrote:
> > On 4/11/06, Mark Hindess wrote:
> >
> >
> >> Personally, obviously,  I'd expect people to run the tests before
> >> committing.
> >>
> >> However, I notice that since enabling the security tests - which fork
> >> for every test - that the tests take over half an hour to run now on
> >> our Linux build machine.  So I can see why enthusiasm might lead to
> >> people not running all the tests but instead perhaps just running
> >> those from the module you are changing.
> >>
> >
> >
> > I think that this message[1] relates to the problem with running big
> test
> > suites. So should we do the similar thing i.e. define which test suites
> > should be run for each module?
> >
> > Thanks,
> > Stepan.
> >
> > [1]
> >
> http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200603.mbox/%3c442C1C79.1060805@sourcelabs.com%3e
> >
> >
> >
>
> Hi Stepan,
>
> We should both define and then automate which test suites get run for
> changes made to a given module. The dependency information is already
> there for us in the contents of each module's MANIFEST.MF file. So, to
> take a recent example which we are all familiar with ;-) , we can see
> that the java.text package exported from the text module is imported by
> modules logging, LUNI, security and SQL.
>
> Folks wishing to verify that changes made to a given module do not break
> other modules should ideally be able to kick off a JUnit run that
> incorporates all of the tests from the other modules into one test
> suite. This could either be folded into the existing "run.tests" target
> of the module's build.xml or else be a separate target like
> "run.depends.tests" (or similar).
> i.e. something like ...
>
> <target name="run.depends.tests">
>
> <mkdir dir="${hy.tests.reports}" />
>
> <junit fork="yes"
> forkmode="once"
> printsummary="withOutAndErr"
> errorproperty="test.error"
> showoutput="on"
> dir="${hy.text.bin.test}"
> jvm="${hy.target}/jre/bin/java">
>
> <jvmarg value="-showversion"/>
>
> <!-- Required by various tests that set security manager etc -->
> <jvmarg
> value="-
> Djava.security.policy=../../../../support/src/test/resources/config/testing.policy
> "
> />
>
> <!-- Required for running the java.net unit tests -->
> <jvmarg
> value="-
> Dtest.ini.file=../../../../support/src/test/resources/config/localhosttest.ini
> "
> />
>
> <env key="JAVA_HOME" value="${hy.target}/jre"/>
>
> <!-- Adding all dependent test bins to the classpath -->
> <classpath>
> <pathelement path="${hy.text.bin.test}"/>
> <pathelement path="${hy.logging.bin.test}"/>
> <pathelement path="${hy.luni.bin.test}"/>
> <pathelement path="${hy.security.bin.test}"/>
> <pathelement path="${hy.sql.bin.test}"/>
> <pathelement path="${hy.tests.support.bin}" />
> </classpath>
>
> <!-- The DependencyTests suite logically groups the other dependent test
> suites -->
> <test name="tests.text.DependencyTests"
> todir="${hy.tests.reports}"
> haltonfailure="no" >
> <formatter type="xml"/>
> </test>
> </junit>
> </target>
>
>
> In this scenario the class tests.text.DependencyTests would be a simple
> logical grouping of the dependent test suites using standard JUnit
> techniques.
> i.e. something like ...
>
> public class DependencyTests {
> public static Test suite() {
> TestSuite suite = new TestSuite("Dependents on java.text module");
> suite.addTestSuite(org.apache.harmony.logging.AllTests.suite());
> suite.addTestSuite(org.apache.harmony.luni.AllTests.suite());
> suite.addTestSuite(org.apache.harmony.security.AllTests.suite());
> suite.addTestSuite(org.apache.harmony.sql.AllTests.suite());
> suite.addTestSuite(org.apache.harmony.logging.AllTests.suite());
> return suite;
> }
> }
>
> ...and each of the AllTests classes referred to were just a logical
> grouping of the test cases in that particular module.
>
>
> I think that such a scheme will be a tremendous convenience to
> developers as it will increase our confidence that breakages introduced
> into other modules by changes in a particular module will be caught soon
> without recourse to running the entire test suite - just the necessary
> ones.
>
> If we see merit in setting up something like this to run alongside the
> existing test targets then I will be happy to open the JIRA and create
> the new Ant targets and test suites.
>
> Best regards,
> George
>
> >
> >> George just commented in a commit message backing out some changes in
> >> text that all the text tests passed even though the changed caused
> >> failures elsewhere.
> >>
> >> Regards,
> >> Mark.
> >>
> >> On 4/11/06, Geir Magnusson Jr <geir@pobox.com> wrote:
> >>
> >>> Just curious (and this isn't a criticism - I'm just as guilty of not
> >>> doing this)...
> >>>
> >>> Don't you run the tests before committing?
> >>>
> >>> geir
> >>>
> >>> George Harley wrote:
> >>>
> >>>> Hi,
> >>>>
> >>>> It *seems* like things started failing after I committed the changes
> >>>>
> >> for
> >>
> >>>> HARMONY-205 last night. I'm looking into this now. If the
> >>>>
> >> investigation
> >>
> >>>> begins to take up too much time I will back the changes out.
> >>>>
> >>>> Best regards,
> >>>> George
> >>>>
> >>>>
> >>>> Stepan Mishura wrote:
> >>>>
> >>>>> The same for me + DatagramChannelTest
> >>>>>
> >>>>> Thanks,
> >>>>> Stepan.
> >>>>>
> >>>>> On 4/11/06, Mark Hindess wrote:
> >>>>>
> >>>>>
> >>>>>> No.  These:
> >>>>>>
> >>>>>> F
> >>>>>>
> >>>>>>
> >>
> org.apache.harmony.security.asn1.der.DerGeneralizedTimeEDTest.testGeneralizedEncoder
> >>
> >>>>>> E
> >>>>>>
> >>>>>>
> >>
> org.apache.harmony.security.asn1.der.DerGeneralizedTimeEDTest.testGeneralizedEncoderDecoder01
> >>
> >>>>>> E
> >>>>>>
> >>>>>>
> >>
> org.apache.harmony.security.asn1.der.DerGeneralizedTimeEDTest.testGeneralizedEncoderDecoder02
> >>
> >>>>>> F org.apache.harmony.security.asn1.der.DerUTCTimeEDTest.testMt
> >>>>>> E
> >>>>>>
> >>>>>>
> >>
> org.apache.harmony.security.x509.PrivateKeyUsagePeriodTest.testEncodeDecode
> >>
> >>>>>> F java.security.cert.X509CertSelectorTest.testSetPrivateKeyValid
> >>>>>> F java.security.cert.X509CertSelectorTest.testMatch
> >>>>>> F java.security.cert.X509CertSelectorTest.testClone
> >>>>>> F tests.api.java.sql.DateTest.testSetTimelong
> >>>>>> F tests.api.java.sql.DateTest.testToString
> >>>>>> F tests.api.java.sql.DateTest.testValueOf
> >>>>>> F tests.api.java.sql.TimestampTest.testSetNanosint
> >>>>>> F tests.api.java.sql.TimestampTest.testToString
> >>>>>> F tests.api.java.util.DateTest.test_toGMTString
> >>>>>> F tests.api.java.util.DateTest.test_toString
> >>>>>>
> >>>>>> Regards,
> >>>>>> Mark.
> >>>>>>
> >>>>>> On 4/11/06, Stepan Mishura <stepan.mishura@gmail.com>
wrote:
> >>>>>>
> >>>>>>
> >>>>>>> On 4/11/06, Mark Hindess wrote:
> >>>>>>>
> >>>>>>>
> >>>>>>>> Yes.  I was using the failureproperty mechanism.  Trying
to get
> >>>>>>>>
> >> this
> >>
> >>>>>>>> property propogated back to the top level ant file was
what I was
> >>>>>>>> having trouble with.
> >>>>>>>>
> >>>>>>>> Using a file as you suggest might help.  I'll give that
a try
> >>>>>>>>
> >>>>>>>>
> >>>>>> shortly...
> >>>>>>
> >>>>>>
> >>>>>>>> Incidentally, I'm seeing 12 failures and 3 errors on
r393111.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>  I guess that you have 9 tests from DatagramChannelTest
passed.
> And
> >>>>>>> 12 +
> >>>>>>>
> >>>>>>>
> >>>>>> 3 =
> >>>>>>
> >>>>>>
> >>>>>>> 15 :-)
> >>>>>>>
> >>>>>>>  (And
> >>>>>>>
> >>>>>>>
> >>>>>>>> there are typos - "mathc" should be "match" - in the
failure
> >>>>>>>>
> >> messages
> >>
> >>>>>>>> for java.security.cert testMatch and testClone.)
> >>>>>>>>
> >>>>>>>>
> >>>>>>> I've fixed typo.
> >>>>>>>
> >>>>>>> Thanks,
> >>>>>>> Stepan.
> >>>>>>>
> >>>>>>> Regards,
> >>>>>>>
> >>>>>>>
> >>>>>>>> Mark.
> >>>>>>>>
> >>>>>>>> On 4/11/06, Stepan Mishura <stepan.mishura@gmail.com>
wrote:
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> On 4/11/06, Mark Hindess wrote:
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> Stepan,
> >>>>>>>>>>
> >>>>>>>>>> I have another build running (but without notifications
going
> to
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>> the
> >>>>>>
> >>>>>>
> >>>>>>>>>> list) that runs:
> >>>>>>>>>>
> >>>>>>>>>> 1) build (with reference jdk)
> >>>>>>>>>> 2) build with what we created with 1)
> >>>>>>>>>> 3) test
> >>>>>>>>>> 4) create classlists and compare with class
load data for
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>> applications
> >>>>>>
> >>>>>>
> >>>>>>>>>>    (tomcat, geronimo, continuum, etc.)
> >>>>>>>>>> 5) generate JAPI results
> >>>>>>>>>>
> >>>>>>>>>> I'd like to fail this build at step 3, but I
can't see an easy
> >>>>>>>>>>
> >> way
> >>
> >>>>>> to
> >>>>>>
> >>>>>>
> >>>>>>>>>> get 'ant -f make/build.xml test' to run all
tests and then fail
> >>>>>>>>>>
> >> if
> >>
> >>>>>> any
> >>>>>>
> >>>>>>
> >>>>>>>>>> of the module test sub-targets had test failures.
 I could
> parse
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>> the
> >>>>>>
> >>>>>>
> >>>>>>>>>> output I suppose, but I'd rather get ant to
propagate the junit
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>> tasks
> >>>>>>
> >>>>>>
> >>>>>>>>>> failure property back up to the top level. 
I've tried a couple
> >>>>>>>>>>
> >> of
> >>
> >>>>>>>>>> things and none seem to work.  Any suggestions
welcome.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>  Mark, did you try failureproperty parameter for
junit task?
> >>>>>>>>> We may add it to ant sub-targets to raise a flag,
for example,
> >>>>>>>>>
> >>>>>>>>>
> >>>>>> create
> >>>>>>
> >>>>>>
> >>>>>>>> file "
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> TESTS.FAILED" in the root,  when tests for some
module fail. And
> >>>>>>>>>
> >> in
> >>
> >>>>>> the
> >>>>>>
> >>>>>>
> >>>>>>>> end
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> of tests suite run check whether this file exists
on not.
> >>>>>>>>>
> >>>>>>>>> Thanks,
> >>>>>>>>> Stepan.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Regards,
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> Mark.
> >>>>>>>>>>
> >>>>>>>>>> On 4/11/06, Stepan Mishura <stepan.mishura@gmail.com>
wrote:
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>> Hi,
> >>>>>>>>>>>
> >>>>>>>>>>> I've checked out at morning last updates,
built the code base
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>> and
> >>>>>>
> >>>>>>
> >>>>>>>> run
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>>> the
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>> tests …and there are 24 tests failures!
> >>>>>>>>>>>
> >>>>>>>>>>> There are 9 tests failures in
> >>>>>>>>>>>
> org.apache.harmony.tests.java.nio.channels.DatagramChannelTest–
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>> I
> >>>>>>
> >>>>>>
> >>>>>>>> saw
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>>> these
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>> failures before from time to time. It seems
that tests depend
> >>>>>>>>>>>
> >> on
> >>
> >>>>>>>> some
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>>> race
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>> conditions because they may pass if I rerun
them. Paulex, are
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>> these
> >>>>>>
> >>>>>>
> >>>>>>>>>> tests
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>> passing for you?
> >>>>>>>>>>>
> >>>>>>>>>>> And there are new 15 test failures.  So
now if I'll make a
> code
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>> update
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>>> or a
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>> bug fix how I can be 100% sure that I don't
do any regression?
> >>>>>>>>>>>
> >>>>>>>>>>> Currently if a commit breaks the Harmony
classlib build a
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>> notification
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>>> with
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>> subject: "[continuum] BUILD FAILURE: Classlib/linux.ia32"
will
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>> be
> >>>>>>
> >>>>>>
> >>>>>>>> send
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>>> to
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>> harmony-commits mailing list. Is it possible
to have the same
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>> for
> >>>>>>
> >>>>>>
> >>>>>>>> tests?
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>>> I
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>> mean that after completing automatic build
the existing
> >>>>>>>>>>>
> >> classlib
> >>
> >>>>>>>> tests
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>>> suite
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>> should be run. If there are failing tests
then a report
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>> notification
> >>>>>>
> >>>>>>
> >>>>>>>>>> with
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>> corresponding subject will be send.
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks,
> >>>>>>>>>>> Stepan Mishura
> >>>>>>>>>>> Intel Middleware Products Division
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>> --
> >>>>>>>>>> Mark Hindess <mark.hindess@googlemail.com>
> >>>>>>>>>> IBM Java Technology Centre, UK.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >> ---------------------------------------------------------------------
> >>
> >>>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >>>>>>>>>> To unsubscribe, e-mail:
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>> harmony-dev-unsubscribe@incubator.apache.org
> >>>>>>
> >>>>>>
> >>>>>>>>>> For additional commands, e-mail:
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>> harmony-dev-help@incubator.apache.org
> >>>>>>
> >>>>>>
> >>>>>>>>> --
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >> ---------------------------------------------------------------------
> >>
> >>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >>>>>>>>> To unsubscribe, e-mail:
> >>>>>>>>>
> >> harmony-dev-unsubscribe@incubator.apache.org
> >>
> >>>>>>>>> For additional commands, e-mail:
> >>>>>>>>>
> >>>>>>>>>
> >>>>>> harmony-dev-help@incubator.apache.org
> >>>>>>
> >>>>>>
> >>>>>>>>> Thanks,
> >>>>>>>>> Stepan Mishura
> >>>>>>>>> Intel Middleware Products Division
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>> --
> >>>>>>>> Mark Hindess <mark.hindess@googlemail.com>
> >>>>>>>> IBM Java Technology Centre, UK.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >> ---------------------------------------------------------------------
> >>
> >>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >>>>>>>> To unsubscribe, e-mail:
> >>>>>>>>
> >> harmony-dev-unsubscribe@incubator.apache.org
> >>
> >>>>>>>> For additional commands, e-mail:
> >>>>>>>>
> >> harmony-dev-help@incubator.apache.org
> >>
> >>>>>>>>
> >>>>>>>>
> >>>>>>> --
> >>>>>>>
> >>>>>>>
> >> ---------------------------------------------------------------------
> >>
> >>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >>>>>>> To unsubscribe, e-mail:
> >>>>>>>
> >> harmony-dev-unsubscribe@incubator.apache.org
> >>
> >>>>>>> For additional commands, e-mail:
> >>>>>>>
> >> harmony-dev-help@incubator.apache.org
> >>
> >>>>>>> Thanks,
> >>>>>>> Stepan Mishura
> >>>>>>> Intel Middleware Products Division
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>> --
> >>>>>> Mark Hindess <mark.hindess@googlemail.com>
> >>>>>> IBM Java Technology Centre, UK.
> >>>>>>
> >>>>>>
> >>>>>>
> >> ---------------------------------------------------------------------
> >>
> >>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >>>>>> To unsubscribe, e-mail:
> harmony-dev-unsubscribe@incubator.apache.org
> >>>>>> For additional commands, e-mail:
> >>>>>>
> >> harmony-dev-help@incubator.apache.org
> >>
> >>>>>>
> >>>>>>
> >>>>> --
> >>>>>
> ---------------------------------------------------------------------
> >>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >>>>> For additional commands, e-mail:
> >>>>>
> >> harmony-dev-help@incubator.apache.org
> >>
> >>>>> Thanks,
> >>>>> Stepan Mishura
> >>>>> Intel Middleware Products Division
> >>>>>
> >>>>>
> >>>>>
> >>>> ---------------------------------------------------------------------
> >>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >>>> For additional commands, e-mail:
> harmony-dev-help@incubator.apache.org
> >>>>
> >>>>
> >>>>
> >>> ---------------------------------------------------------------------
> >>> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >>>
> >>>
> >>>
> >> --
> >> Mark Hindess <mark.hindess@googlemail.com>
> >> IBM Java Technology Centre, UK.
> >>
> >> ---------------------------------------------------------------------
> >> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >>
> >>
> >>
> >
> >
> > --
> > ---------------------------------------------------------------------
> > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >
> > Thanks,
> > Stepan Mishura
> > Intel Middleware Products Division
> >
> >
>
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>


--
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org

Thanks,
Stepan Mishura
Intel Middleware Products Division

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message