Mailing-List: contact user-help@jmeter.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "JMeter Users List" <user@jmeter.apache.org>
Received-SPF: pass (athena.apache.org: domain of sebbaz@gmail.com designates
 209.85.212.169 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <9B2F95EE-308D-448F-B51D-8BDE5167C7F6@azulsystems.com>
References: 
 <CAOGo0VY4sKMOHCV=PJG8mNWf+ADhuqKK2J3AB+g4sVvcgQqVFg@mail.gmail.com>
	<9B2F95EE-308D-448F-B51D-8BDE5167C7F6@azulsystems.com>
Date: Sat, 19 Oct 2013 12:45:01 +0100
Message-ID: 
 <CAOGo0VY0Zo0a9JHFsrasRXibenxmK0rAGPxsC_VwBJizC_SLmg@mail.gmail.com>
Subject: Re: Coordinated Omission - detection and reporting ONLY
From: sebb <sebbaz@gmail.com>
To: JMeter Users List <user@jmeter.apache.org>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On 19 October 2013 02:17, Gil Tene <gil@azulsystems.com> wrote:
>
>> [Trying again - please do not hijack this thread.]
>>
>> The Constant Throughput Timer (CTT) calculates the desired wait time,
>> and if this is less than zero - i.e. a sample should already have been
>> generated - it could trigger the creation of a failed Assertion (or simi=
lar)
>> showing the time difference.

N.B.     ^^^^^^^^^^^

>>
>> Would this be sufficient to detect all CO occurrences?
>
> Two issues:
>
> 1. It would detect that CO probably happened, but not how much of it happ=
ened, Missing 1msec or 1 minute will look the same.

Huh? The time difference would be reported - is that not enough?

> 2. It would only detect CO in test plans that include an actual CTT (Cons=
tant Throughput Timer). It won't work for other timers, or when no timers a=
re used

Indeed, but in such cases is there any indication of the expected
transaction rate?

>> If not, what other metric needs to be checked?
>
> There are various things you can look for.
>
> Our OutlierCorrector work includes a pretty elaborate CO detector. It sit=
s either inline on the listener notification stream, or parses the log file=
. The detector identifies sampler patterns, establishes expected interval b=
etween patterns, and detects when the actual interval falls far above the e=
xpected interval. This is a code change to JMeter, but a pretty localized o=
ne.

AIUI CO happens when the sample takes much longer than usual, thus
(potentially) delaying any subseqent samples.
Overlong samples can already be detected using a Delay Assertion.

>> Even if it is not the only possible cause, would it be useful as a
>> starting point?
>
> Yes. As a warning flag saying "throw away these test results".

The results are not necessarily useless; at the very least they will
show the load at which the system is starting to slow down.

>> I am assuming that the CTT is the primary means of controlling the
>> sample request rate.
>
> Unfortunately many test scenarios I've seen use other means. Many people =
use other timers or other means for think time emulation.

The CTT is not intended for think time emulation. It is intended to
ensure a specific transaction rate.

Think time obviously affects the transaction rate, but is not the sole
determinant.

>> If there are other elements that are commonly used to control the
>> rate, please note them here.
>>
>> N.B: this thread is only for discussion of how to detect CO and how to
>> report it.
>
> Reporting the existence of CO is an interesting starting point. But the o=
nly right way to deal with such a report showing the existence of CO (with =
no magnitude or other metrics) is to say " I guess the data I got is comple=
te crap, so all the stats and graphs I'm seeing mean nothing".

I disagree that the output is useless.

The delays are reported, so one can see how badly the test was
affected. If the delays are all small, then a slight adjustment of the
thread count and transaction rate should eliminate them.

Only if the delays are large are the results less meaningful, though
they can still show the base load at which the server starts to slow
down.

> If you can report "how much" CO you saw,

As I wrote originally, the CTT would report the time diffence (i.e. delay).

> it may help a bit in determining how bad the data is, and how the stats s=
hould be treated by the reader. E.g. if you know that CO totaling some amou=
nt of time X in a test of length Y had occured, then you know that any perc=
entile above (100 * (1-X)/Y) is completely bogus, and should be assumed to =
be equal to the experienced max value. You can also take the approach that =
the the rest of the percentiles should be shifted down by at least  (100 * =
X / Y). e.g. If you had CO that covered only 0.01% of the total test time, =
that would be relatively good news.

Exactly.

> But if you had CO covering 5% of the test time, your measured 99%'ile is =
actually the 94%'ile]. Averages are unfortunately anyone's guess when CO is=
 in play and not actually corrected for.
>
> Once you detect both the existence and the magnitude of CO, correcting fo=
r it is actually pretty easy. The detection of "how much" is the semi-hard =
part.

Detecting the delay using the CTT is trivial; it already has to
calculate it to decide how long to wait. A negative wait is obviously
a missed start time.
Or am I missing something here?

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
For additional commands, e-mail: user-help@jmeter.apache.org