harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stepan Mishura" <stepan.mish...@gmail.com>
Subject Re: [general][testing] Increase timeout or split into sub-suites (and how)? (was: Re: [jira] Commented: (HARMONY-3319) [jdktools][jpda] JDWP unit tests fail with timeout)
Date Tue, 13 Mar 2007 04:19:17 GMT
On 3/12/07, Ivan Popov wrote:
>
> Stepan,
>
> I agree that splitting tests into separate <batchtest> items and using
> 'perBatch' mode is better than running several junit tasks and using
> macros. However, it would be better to specify this as a default mode
> for all unit tests, but not hardcoding it as an exception for one
> particular module, using not so evident rules for setting junit fork
> mode. In case of local changes using separate junit tasks looks more
> clear and understandable.
>
> I vote for changing default fork mode for junit tests to 'perBatch',
> this enables flexible way to control tests execution and optimize
> usage of junit timeout. I don't think that increasing default timeout
> value is a good idea. This may lead to that a single hung test will
> significantly increase tests execution time, which is especially
> unwanted in automatic tests runs.


Agreed for changing default forkmode to 'perBatch'.

-Stepan.

Thanks.
> Ivan
>
> On 3/12/07, Stepan Mishura <stepan.mishura@gmail.com> wrote:
> > Hi Ivan,
> >
> > I think that the issue is common for all testing suites so I'd like to
> hear
> > other opinions.
> >
> > Summary: junit timeout works for per forked VM not for an individual
> test in
> > a test suite.
> > For example, if each test in a test suite fits in timeout but overall
> time
> > for testsuite exceeds timeout then testsuite passes when
> forkmode=perTest
> > and fails with timeout when forkmode=once. (I think that is possible for
> any
> > classlib module as far as the number of tests is growing).
> >
> > Currently we had such situation for JDPA tests[1] - the test suite run
> did't
> > fit into timeout so JPDA test suite failed if forkmode=once. The most
> simple
> > and quick solution was to increase timeout (current timeout=15min).
> Another
> > way was to split the test suite into sub-suites that was suggested by
> the
> > JIRA patch (by adding a macrodef to run two junit tasks). I think that
> > timeout value should have reasonable value (not too big) so I desided to
> > split the testsuite but did it in different way - I created two
> batchtest
> > junit elements (to split the suite) and run tests in forkmode=perBatch.
> >
> > Are there opinions about:
> > 1) Reasonable value for timeout? Should we define any limit for it?
> > 2) Clear and understandable way for splitting the suite: macrodef with
> > several junit tasks OR one junit task and several batchtests elements
> > within.
> >
> > [1] https://issues.apache.org/jira/browse/HARMONY-3319
> >
> > Thanks,
> > Stepan.
> >
> > On 3/9/07, Ivan Popov (JIRA) <jira@apache.org> wrote:
> > >
> > >
> > >    [
> > >
> https://issues.apache.org/jira/browse/HARMONY-3319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12479590
> ]
> > >
> > > Ivan Popov commented on HARMONY-3319:
> > > -------------------------------------
> > >
> > > I don't think that your solution with adding special property "
> > > hy.jpda.test.forkmode" which value is calculated depending on the
> value of
> > > standard property "hy.test.forkmode" according to not trivial rules:
> > >
> > >        <!-- run JPDA tests with forkmode=perBatch if common
> forkmode=once
> > > -->
> > >        <condition property="hy.jpda.test.forkmode" value="perBatch"
> > > else="perTest">
> > >            <equals arg1="${hy.test.forkmode}" arg2="once" />
> > >        </condition>
> > >
> > > is better than was proposed in the patch.
> > >
> > > You still have to divide tests into two groups, but also introduce not
> so
> > > evident calculation of a special property, which one need to take into
> > > account. I think the previous suggestion with dividing tests into two
> junit
> > > tasks is more clear and understandable.
> > >
> > >
> > > > [jdktools][jpda] JDWP unit tests fail with timeout
> > > > --------------------------------------------------
> > > >
> > > >                 Key: HARMONY-3319
> > > >                 URL:
> https://issues.apache.org/jira/browse/HARMONY-3319
> > > >             Project: Harmony
> > > >          Issue Type: Bug
> > > >          Components: JDK
> > > >         Environment: Windows/ia32, Linux/ia32, Harmony-jdk-r515478
> > > >            Reporter: Ivan Popov
> > > >         Assigned To: Stepan Mishura
> > > >         Attachments: jpda_build.patch
> > > >
> > > >
> > > > JDWP unit tests from JPDA component started with ant build usually
> fail
> > > with timeout:
> > > > <...>
> > > >     [junit] TEST
> > > org.apache.harmony.jpda.tests.jdwp.ReferenceType.GetValues003TestFAILED
> > > >     [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed:
> 9.578sec
> > > >     [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed:
> 9.543sec
> > > >     [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed:
> 9.537sec
> > > >     [junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 0
> sec
> > > >     [junit] Tests FAILED (timeout)
> > > > <...>
> > > > full-report:
> > > > [junitreport] [Fatal Error] :-1:-1: Premature end of file.
> > > > [junitreport] The file
> > > .../trunk/working_jdktools/build/test_report/TEST-
> > > org.apache.harmony.jpda.tests.jdwp.ReferenceType.InterfacesTest.xml is
> not
> > > a valid XML document. It is possibly corrupted.
> > > > The problem is in default timeout (120000 milliseconds, or 15
> minutes)
> > > specified for junit task in 'timeout" option:
> > > >                timeout="${hy.test.timeout}"
> > > > JUnit task reference reads for this option:
> > > >   Cancel the individual tests if they don't finish in the given time
> > > (measured in milliseconds). Ignored if fork is disabled.
> > > > However, it seems that this option is applied to the whole tests set
> > > specified in <batchtest> item.  All JDWP tests are specified in a
> single
> > > <batchtest> and their execution usually exceeds 15 minutes, so junit
> task
> > > cancels them with timeout.
> > > > There are several "long-running" tests, each takes about 1 minute.
> These
> > > tests can be moved to a separate <batchtest> item  to satisfy the
> default
> > > timeout.
> > >
> > > --
> > > This message is automatically generated by JIRA.
> > > -
> > > You can reply to this email to add a comment to the issue online.
> > >
> > >
> >
> >
> > --
> > Stepan Mishura
> > Intel Enterprise Solutions Software Division
> >
>



-- 
Stepan Mishura
Intel Enterprise Solutions Software Division

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message