jmeter-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robin D. Wilson" <rwils...@gmail.com>
Subject RE: Display total execution time for test plan
Date Wed, 21 Mar 2012 13:50:34 GMT
I think it depends a lot on what you are testing. If you are trying to benchmark system performance,
total test duration can be a good indicator (and a quick-glance check) of system performance.
For example, my performance benchmarks are configured to run (without ramp up) between 10
and 300 threads (depending on the test), in such a way as to guarantee that I am exercising
the system at near capacity (for each benchmark). Because I am running the benchmarks for
each release of our system, I have a history of the test performance.

The total test duration is a good "overall" measure of the performance of any given benchmark.
And it is what I used to figure out that JMeter wasn't properly reporting the 'average' sample
times - until I changed my config to use the following settings:

	sampleresult.useNanoTime=false
	sampleresult.nanoThreadSleep=0

I was seeing the same 'average' times, but the total execution time for a thread group was
increasing with each successive new release of code. This suggested that something was slowing
things down in my code-base. After I made the above config change to JMeter (2.6) I could
see that the average sample times were actually much higher than my benchmarks had been recording.

Without being able to explicitly see the execution duration times (or using the average sample
times to calculate the test duration), I would have missed the fact that my benchmarks were
getting worse.

--
Robin D. Wilson
Sr. Director of Web Development
KingsIsle Entertainment, Inc.
VOICE: 512-777-1861
www.KingsIsle.com

-----Original Message-----
From: Adrian Speteanu [mailto:asp.adieu@gmail.com] 
Sent: Wednesday, March 21, 2012 6:03 AM
To: JMeter Users List
Subject: Re: Display total execution time for test plan

Hi,

I suspect you weren't interested in start of / end of test. But usually
this is how you get total test time :).

It doesn't make sense to have a test that gradually starts 1000 users and
test stops when all finished their planned sessions. Its not even useful to
measure how long the test took.

Why: no live application works likes this in production conditions. At the
beginning and end of test you have less than 1000 users logged in. What if
the ramp up of the 1000 threads affect average results, or even total
execution time?

Check out Sergio's reply. You simulate what users do - true, but at macro
level, and you design your test plan in such a manner to respect your
requirements: 1000 sessions logged in and a maximum of 10 hits / s. When
you have such a test, than you check out the statistics from Aggregate
Graph, Summary Report + make some nice graphs with some of the cooler
things that you monitor. Don't forget CPU, RAM, Network usage on the server
side. That's what you measure and compare before and after a change.

And if a change affects a particular request, focus measurements and
reporting on that specific request.

Its good to know what one user does, but its better to know what workload
your app receives:
  - 1000 logged in, unique and active sessions
  - 80% make page views in section X
  - 10% use the forum (or whatever)
  - 1% upload files
  - 2% download stuff during their session
 ....
etc - this is just an example...

If you get this right for your particular application, then you need to
measure the statistics of the response time: avg, median, 90th line. See
how they evolve during the test (this is even better than looking at the
values for the entire period) and so on. But all this makes measuring total
time very irrelevant in 90% of tests or more.

Adrian

On Wed, Mar 21, 2012 at 12:06 AM, sergio <sergio@bosoconsulting.it> wrote:

> Hi Adrian and Eric,
>
> maybe I'm missing some point, but to me the total duration of the test is
> rarely important nor predictable.
>
> If you need it as a baseline, you can use an aggregate result listener,
> run some test (maybe with one or two users) and then
> you can multiply the number of samples (eventually divided the number of
> loops executed) by the average execution time.
> So you can easily have the net time you need to do a single loop.
> This is net of time spent on timers.
>
> But when you start having 1000 users, you have a lot of parallelizaton,
> but obviously not the 100% (that would be ideal).
> Also in some case, you have to add the ramp-up time.
>
> In my experience, we usually end up measuring the behaviour of few key
> transactions (e.g. submit the order, or login/logout), under different
> situations and loads,
> The relationship between average, mean, 90nth % and max return an idea of
> the way things go.
> Note that these transactions are also the longest.
>
> A static page or an image takes few msec to download, and most of the time
> spent is due to the network latency,
> which is not something we can easily optimize.
>
> This is my point of view, feel free to share your thoughts.
> best regards
>
> Sergio
>
> Il 20/03/2012 17:55, Erik Pragt ha scritto:
>
>  Hi Adrian,
>>
>> Thanks for the super quick reply. I'm a bit surprised by your first
>> remark though, so maybe I'm having a wrong approach here.
>>
>> I'm currently developing an application which might have some
>> performance issues. Our current target is around 1000 simultaneous
>> logged in users, and around 10 concurrent 'clicks'. My current
>> approach was to sort of simulate that behavior in JMeter, check how
>> long it takes for the simulated users to finish their flows, make some
>> adjustments, test again, and check if my simulated users are faster
>> than before. Based on this, I need the total execution time, but
>> apparently this is not the usual approach, else it would certainly
>> have been in there somewhere.
>>
>> Could you recommend what would be a better way to test my scenario?
>> I'm not a performance rock star at all, so I'm very curious what would
>> be an effective way in improving the application and using JMeter as
>> the load generator in that.
>>
>> Kind regards,
>>
>> Erik Pragt
>>
>> On Tue, Mar 20, 2012 at 5:44 PM, Adrian Speteanu<asp.adieu@gmail.com>
>>  wrote:
>>
>>> Hi Erik,
>>>
>>> A very interesting idea.
>>>
>>> You can find start / stop time in jmeter's log. When running from a
>>> console
>>> in non-gui mode, you also get some more statistics then in GUI (how long
>>> the test ran). You can also schedule a test to run for a certain amount
>>> of
>>> time, or starting / stopping at certain hours (so you don't have to worry
>>> about this stuff).
>>>
>>> If you are interested in response times, however, the sum of all
>>> requests,
>>> then things get more complicated.
>>>
>>> Adrian
>>>
>>> On Tue, Mar 20, 2012 at 6:21 PM, Erik Pragt<erik.pragt@jworks.nl>
>>>  wrote:
>>>
>>>  Hi all,
>>>>
>>>> I've created a test plan to put some load on a flow of pages we have.
>>>> I'm quite new to JMeter, and I have a small question on how to get the
>>>> information I'm looking for. I've got a working test plan, I can see
>>>> the samples, the throughput, etc, but I can't find anywhere what the
>>>> time was to execute this testplan, or a single loop of this testplan
>>>> when I execute it multiple times.
>>>>
>>>> Can someone give me a small heads up how I can record and view this
>>>> time?
>>>>
>>>> Kind regards,
>>>>
>>>> Erik Pragt
>>>>
>>>> ------------------------------**------------------------------**
>>>> ---------
>>>> To unsubscribe, e-mail: user-unsubscribe@jmeter.**apache.org<user-unsubscribe@jmeter.apache.org>
>>>> For additional commands, e-mail: user-help@jmeter.apache.org
>>>>
>>>>
>>>>  ------------------------------**------------------------------**
>> ---------
>> To unsubscribe, e-mail: user-unsubscribe@jmeter.**apache.org<user-unsubscribe@jmeter.apache.org>
>> For additional commands, e-mail: user-help@jmeter.apache.org
>>
>>
>
> --
>
> Ing. Sergio Boso
>
> In caso di erronea ricezione da parte di persona diversa, siete pregati di
> eliminare il messaggio e i suoi allegati in modo definitivo dai vostri
> archivi e di volercelo comunicare immediatamente restituendoci il messaggio
> via e-mail al seguente indirizzosergio@**bosoconsulting.it<indirizzosergio@bosoconsulting.it><mailto:
> sergioboso@yahoo.it>
> L’interessato può, inoltre, esercitare tutti i diritti di accesso sui
> propri dati previsti dal decreto 196/2003, tra i quali i diritti di
> rettifica, aggiornamento e cancellazione, inviando un messaggio all’
> indirizzo:sergio@**bosoconsulting.it<indirizzo%3Asergio@bosoconsulting.it><mailto:
> sergioboso@yahoo.it>
>
>
>
>
>
>
> ------------------------------**------------------------------**---------
> To unsubscribe, e-mail: user-unsubscribe@jmeter.**apache.org<user-unsubscribe@jmeter.apache.org>
> For additional commands, e-mail: user-help@jmeter.apache.org
>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
For additional commands, e-mail: user-help@jmeter.apache.org


Mime
View raw message