jmeter-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From bugzi...@apache.org
Subject [Bug 60725] New: Allowing for slow-running test systems
Date Sat, 11 Feb 2017 13:46:58 GMT
https://bz.apache.org/bugzilla/show_bug.cgi?id=60725

            Bug ID: 60725
           Summary: Allowing for slow-running test systems
           Product: JMeter
           Version: unspecified
          Hardware: All
                OS: All
            Status: NEW
          Severity: minor
          Priority: P2
         Component: Main
          Assignee: issues@jmeter.apache.org
          Reporter: sebb@apache.org
  Target Milestone: ---

Some of the test cases make assumptions about how long a Thread.sleep() will
take.

However this can vary between systems due to timer granularity and how busy the
system is.

If the assumptions are wrong, then the test may fail unless the variation is
allowed for.

One way to do this is to allow a margin of error in the assertion.

However if a system runs very slowly then the error margin may have to be
increased so much for the test to pass that it no longer catches broken code.

This is/was the approach taken in many of the unit tests.

An alternative is to measure the actual sleep times, and calculate the expected
result accordingly.

This is what has been done in

http://svn.apache.org/viewvc?rev=1782600&view=rev

and

 http://svn.apache.org/viewvc?rev=1782321&view=rev

Note that not all Thread.sleep() instances affect the outcome of tests.
Care must be taken to ensure that the test will still fail if the code is
broken, i.e. which sleeps are relevant to the test result.

Other sleeps may be necessary to ensure the test is working, but may not
directly affect the result.

In the case of TestSampleResult.testPauseTrue(), two of the sleeps represent
the sampler doing work, and another is the pause time where the sampler is not
supposed to record the time.

All 3 sleeps are necessary, but the pause time is not directly relevant as long
as it is long enough to ensure that the test would fail if the sampler
calculation were incorrect. e.g. if the pause time was too short (or zero) then
the test would not show that the sampler was correctly ignoring the time spent
in the paused state.

I fixed the two recent examples where recent CI tests failed, but there are
probably some more lurking.

-- 
You are receiving this mail because:
You are the assignee for the bug.
Mime
View raw message