harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stepan Mishura" <stepan.mish...@gmail.com>
Subject [general][testing] Increase timeout or split into sub-suites (and how)? (was: Re: [jira] Commented: (HARMONY-3319) [jdktools][jpda] JDWP unit tests fail with timeout)
Date Mon, 12 Mar 2007 05:52:15 GMT
Hi Ivan,

I think that the issue is common for all testing suites so I'd like to hear
other opinions.

Summary: junit timeout works for per forked VM not for an individual test in
a test suite.
For example, if each test in a test suite fits in timeout but overall time
for testsuite exceeds timeout then testsuite passes when forkmode=perTest
and fails with timeout when forkmode=once. (I think that is possible for any
classlib module as far as the number of tests is growing).

Currently we had such situation for JDPA tests[1] - the test suite run did't
fit into timeout so JPDA test suite failed if forkmode=once. The most simple
and quick solution was to increase timeout (current timeout=15min). Another
way was to split the test suite into sub-suites that was suggested by the
JIRA patch (by adding a macrodef to run two junit tasks). I think that
timeout value should have reasonable value (not too big) so I desided to
split the testsuite but did it in different way - I created two batchtest
junit elements (to split the suite) and run tests in forkmode=perBatch.

Are there opinions about:
1) Reasonable value for timeout? Should we define any limit for it?
2) Clear and understandable way for splitting the suite: macrodef with
several junit tasks OR one junit task and several batchtests elements
within.

[1] https://issues.apache.org/jira/browse/HARMONY-3319

Thanks,
Stepan.

On 3/9/07, Ivan Popov (JIRA) <jira@apache.org> wrote:
>
>
>    [
> https://issues.apache.org/jira/browse/HARMONY-3319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12479590]
>
> Ivan Popov commented on HARMONY-3319:
> -------------------------------------
>
> I don't think that your solution with adding special property "
> hy.jpda.test.forkmode" which value is calculated depending on the value of
> standard property "hy.test.forkmode" according to not trivial rules:
>
>        <!-- run JPDA tests with forkmode=perBatch if common forkmode=once
> -->
>        <condition property="hy.jpda.test.forkmode" value="perBatch"
> else="perTest">
>            <equals arg1="${hy.test.forkmode}" arg2="once" />
>        </condition>
>
> is better than was proposed in the patch.
>
> You still have to divide tests into two groups, but also introduce not so
> evident calculation of a special property, which one need to take into
> account. I think the previous suggestion with dividing tests into two junit
> tasks is more clear and understandable.
>
>
> > [jdktools][jpda] JDWP unit tests fail with timeout
> > --------------------------------------------------
> >
> >                 Key: HARMONY-3319
> >                 URL: https://issues.apache.org/jira/browse/HARMONY-3319
> >             Project: Harmony
> >          Issue Type: Bug
> >          Components: JDK
> >         Environment: Windows/ia32, Linux/ia32, Harmony-jdk-r515478
> >            Reporter: Ivan Popov
> >         Assigned To: Stepan Mishura
> >         Attachments: jpda_build.patch
> >
> >
> > JDWP unit tests from JPDA component started with ant build usually fail
> with timeout:
> > <...>
> >     [junit] TEST
> org.apache.harmony.jpda.tests.jdwp.ReferenceType.GetValues003Test FAILED
> >     [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 9.578sec
> >     [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 9.543sec
> >     [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 9.537sec
> >     [junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 0 sec
> >     [junit] Tests FAILED (timeout)
> > <...>
> > full-report:
> > [junitreport] [Fatal Error] :-1:-1: Premature end of file.
> > [junitreport] The file
> .../trunk/working_jdktools/build/test_report/TEST-
> org.apache.harmony.jpda.tests.jdwp.ReferenceType.InterfacesTest.xml is not
> a valid XML document. It is possibly corrupted.
> > The problem is in default timeout (120000 milliseconds, or 15 minutes)
> specified for junit task in 'timeout" option:
> >                timeout="${hy.test.timeout}"
> > JUnit task reference reads for this option:
> >   Cancel the individual tests if they don't finish in the given time
> (measured in milliseconds). Ignored if fork is disabled.
> > However, it seems that this option is applied to the whole tests set
> specified in <batchtest> item.  All JDWP tests are specified in a single
> <batchtest> and their execution usually exceeds 15 minutes, so junit task
> cancels them with timeout.
> > There are several "long-running" tests, each takes about 1 minute. These
> tests can be moved to a separate <batchtest> item  to satisfy the default
> timeout.
>
> --
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>
>


-- 
Stepan Mishura
Intel Enterprise Solutions Software Division

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message