jmeter-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vladimir Sitnikov <sitnikov.vladi...@gmail.com>
Subject Re: Throughput timers vs Poisson vs etc
Date Fri, 06 Feb 2015 20:32:21 GMT
> Did you try Constant Throughput Timer

I did, however it does have drawbacks.
1.1) It does not always work. Sorry, it was long ago I've last tested
it, so I cannot list lots of examples from the top of my head.
For instance, I've just created 100 threads, set thread duration to be
20 seconds and put constant throughput timer of 60 per minute.
I expect exactly 20 items of "debug sampler" to appear, but 100 happens.
Test plan:
ThreadGroup(100 threads, 20 seconds test duration)
  Loop(Forever)
    DebugSampler(for_throughput)
      ConstantThroughputTimer(60 per minute)
    DebugSampler(work)

1.2) It tries to adapt to the target based on the duration of the loop
execution.
It works really awful for long iterations. For instance, if a single
iteration takes 10 minutes and I want 10 per hour, then CTT has just
no time to "adapt".
If an iteration takes longer than CTT expected, then you're screwed.
You get less executions than required.

1.3) CTT is not reproducible. See point 3 of original email on
"Repeatable test profile".

1.4) CTT tries to do even delay between iterations. This plays awful
when you have multiple scenarios running at once if the scenarios have
the same target throughput. You end up with sin(t) kind of CPU%.

5) "Bursty load without explicit batches". If I set CTT at 1 per
minute, it will never launch iterations closer than 60 seconds
(assuming all the bugs like 1.1 are fixed). However, in reality
arrivals often have spikes. For instance, in poisson process (those
kind of arrivals are common to happen in telecom industry)
inter-arrival durations vary (bingo!).
In other words, if an action takes 30 seconds to execute and CTT
launches it every 60 seconds, then this kind of testing will NEVER
catch a case when two actions run concurrently. With inter-arrival
delays like 10 sec, 110 sec, 80 sec, 40 sec throughput would still
result in "1 per minute" on average, but it will catch more
performance bugs (e.g. the one when two concurrent actions result in
out of memory).

> Yes you can, just schedule them :
> Scheduler > Startup Delay + Duration

It does not scale: suppose you have 20 thread groups (e.g. 20 UI scenarios).
How would you manage all those durations?
How would you change the duration if throughput needs to be changed?
(hint: in scalability tests throughput is often need to be altered for
each run)
How would you pass the script to your buddy?

Even if you could configure it once, those 0:0:0 defaults will
definitely bite you very soon (as a new group is added).

Vladimir

Mime
View raw message