harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ivan Popov" <ivan.g.po...@gmail.com>
Subject Re: [general][testing] Increase timeout or split into sub-suites (and how)? (was: Re: [jira] Commented: (HARMONY-3319) [jdktools][jpda] JDWP unit tests fail with timeout)
Date Mon, 12 Mar 2007 10:03:17 GMT
Stepan,

I agree that splitting tests into separate <batchtest> items and using
'perBatch' mode is better than running several junit tasks and using
macros. However, it would be better to specify this as a default mode
for all unit tests, but not hardcoding it as an exception for one
particular module, using not so evident rules for setting junit fork
mode. In case of local changes using separate junit tasks looks more
clear and understandable.

I vote for changing default fork mode for junit tests to 'perBatch',
this enables flexible way to control tests execution and optimize
usage of junit timeout. I don't think that increasing default timeout
value is a good idea. This may lead to that a single hung test will
significantly increase tests execution time, which is especially
unwanted in automatic tests runs.

Thanks.
Ivan

On 3/12/07, Stepan Mishura <stepan.mishura@gmail.com> wrote:
> Hi Ivan,
>
> I think that the issue is common for all testing suites so I'd like to hear
> other opinions.
>
> Summary: junit timeout works for per forked VM not for an individual test in
> a test suite.
> For example, if each test in a test suite fits in timeout but overall time
> for testsuite exceeds timeout then testsuite passes when forkmode=perTest
> and fails with timeout when forkmode=once. (I think that is possible for any
> classlib module as far as the number of tests is growing).
>
> Currently we had such situation for JDPA tests[1] - the test suite run did't
> fit into timeout so JPDA test suite failed if forkmode=once. The most simple
> and quick solution was to increase timeout (current timeout=15min). Another
> way was to split the test suite into sub-suites that was suggested by the
> JIRA patch (by adding a macrodef to run two junit tasks). I think that
> timeout value should have reasonable value (not too big) so I desided to
> split the testsuite but did it in different way - I created two batchtest
> junit elements (to split the suite) and run tests in forkmode=perBatch.
>
> Are there opinions about:
> 1) Reasonable value for timeout? Should we define any limit for it?
> 2) Clear and understandable way for splitting the suite: macrodef with
> several junit tasks OR one junit task and several batchtests elements
> within.
>
> [1] https://issues.apache.org/jira/browse/HARMONY-3319
>
> Thanks,
> Stepan.
>
> On 3/9/07, Ivan Popov (JIRA) <jira@apache.org> wrote:
> >
> >
> >    [
> > https://issues.apache.org/jira/browse/HARMONY-3319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12479590]
> >
> > Ivan Popov commented on HARMONY-3319:
> > -------------------------------------
> >
> > I don't think that your solution with adding special property "
> > hy.jpda.test.forkmode" which value is calculated depending on the value of
> > standard property "hy.test.forkmode" according to not trivial rules:
> >
> >        <!-- run JPDA tests with forkmode=perBatch if common forkmode=once
> > -->
> >        <condition property="hy.jpda.test.forkmode" value="perBatch"
> > else="perTest">
> >            <equals arg1="${hy.test.forkmode}" arg2="once" />
> >        </condition>
> >
> > is better than was proposed in the patch.
> >
> > You still have to divide tests into two groups, but also introduce not so
> > evident calculation of a special property, which one need to take into
> > account. I think the previous suggestion with dividing tests into two junit
> > tasks is more clear and understandable.
> >
> >
> > > [jdktools][jpda] JDWP unit tests fail with timeout
> > > --------------------------------------------------
> > >
> > >                 Key: HARMONY-3319
> > >                 URL: https://issues.apache.org/jira/browse/HARMONY-3319
> > >             Project: Harmony
> > >          Issue Type: Bug
> > >          Components: JDK
> > >         Environment: Windows/ia32, Linux/ia32, Harmony-jdk-r515478
> > >            Reporter: Ivan Popov
> > >         Assigned To: Stepan Mishura
> > >         Attachments: jpda_build.patch
> > >
> > >
> > > JDWP unit tests from JPDA component started with ant build usually fail
> > with timeout:
> > > <...>
> > >     [junit] TEST
> > org.apache.harmony.jpda.tests.jdwp.ReferenceType.GetValues003Test FAILED
> > >     [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 9.578sec
> > >     [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 9.543sec
> > >     [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 9.537sec
> > >     [junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 0 sec
> > >     [junit] Tests FAILED (timeout)
> > > <...>
> > > full-report:
> > > [junitreport] [Fatal Error] :-1:-1: Premature end of file.
> > > [junitreport] The file
> > .../trunk/working_jdktools/build/test_report/TEST-
> > org.apache.harmony.jpda.tests.jdwp.ReferenceType.InterfacesTest.xml is not
> > a valid XML document. It is possibly corrupted.
> > > The problem is in default timeout (120000 milliseconds, or 15 minutes)
> > specified for junit task in 'timeout" option:
> > >                timeout="${hy.test.timeout}"
> > > JUnit task reference reads for this option:
> > >   Cancel the individual tests if they don't finish in the given time
> > (measured in milliseconds). Ignored if fork is disabled.
> > > However, it seems that this option is applied to the whole tests set
> > specified in <batchtest> item.  All JDWP tests are specified in a single
> > <batchtest> and their execution usually exceeds 15 minutes, so junit task
> > cancels them with timeout.
> > > There are several "long-running" tests, each takes about 1 minute. These
> > tests can be moved to a separate <batchtest> item  to satisfy the default
> > timeout.
> >
> > --
> > This message is automatically generated by JIRA.
> > -
> > You can reply to this email to add a comment to the issue online.
> >
> >
>
>
> --
> Stepan Mishura
> Intel Enterprise Solutions Software Division
>

Mime
View raw message