Mailing-List: contact jmeter-user-help@jakarta.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "JMeter Users List" <jmeter-user@jakarta.apache.org>
Received-SPF: pass (nike.apache.org: domain of rwilson2@gmail.com designates
 209.85.213.172 as permitted sender)
From: "Robin D. Wilson" <rwilson2@gmail.com>
To: "'JMeter Users List'" <jmeter-user@jakarta.apache.org>
References: <005901cc717e$85bdacc0$91390640$@gmail.com>
 <CA94C53B.DD5E%olivere@datacom.co.nz>
In-Reply-To: <CA94C53B.DD5E%olivere@datacom.co.nz>
Subject: RE: Flaw in how JMeter runs threads...
Date: Mon, 12 Sep 2011 16:20:49 -0500
Message-ID: <006d01cc7191$d90323c0$8b096b40$@gmail.com>
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: 7bit
thread-index: AQEP1voThTQTQj8bE6ZcO94lWmweUZbDczKw
Content-Language: en-us

I am doing a baseline, and I am trying to repeat the same test... As I said,
it is only a 'small'  flaw - nothing serious.

That being said, it is also reflecting a perspective on testing that is
perhaps different than yours. When I look at the results, I'd like them to
mean something, and I'm just pointing out that they mean something different
than I prefer right now. Having a ramp up and ramp down period may be
appropriate for your tests, but it is skewing my results rather
significantly. I would say that I am seeing about a 10% difference in
'average duration' times caused by this small issue.

I'm not suggesting that other tests aren't useful or productive - just that
I would _like_ to see one that maintains a constant number of threads
through the end of the test.

BTW, I can accommodate the 'ramp up' period by starting a test - letting it
run for a few seconds - killing it, clearing the results and starting it
again... Then second run is pretty much an instantaneous ramp up (with minor
effect on the results). But the ramp-down is a lot harder to control, and
therefore is what I'm discussing. The ramp up period is less than 3 seconds,
the ramp-down period may be over a minute right now - and that is why my
results are being skewed.

A constant throughput timer will certainly 'fix' my results, but it won't
test what I'm trying to test either...

--
Robin D. Wilson
Sr. Director of Web Development
KingsIsle Entertainment, Inc.
VOICE: 512-777-1861
www.KingsIsle.com


-----Original Message-----
From: Oliver Erlewein [DATACOM] [mailto:Oliver.Erlewein@datacom.co.nz] 
Sent: Monday, September 12, 2011 3:37 PM
To: JMeter Users List
Subject: Re: Flaw in how JMeter runs threads...

The simple answer is you would never do that. You will always have a ramp-up
and a ramp-down. You should exclude these phases in your calculations if you
need the values under load only. Even if you "thread issue" were fixed, how
would you deal with differing response times? They still affect your load.

If you use a constant throughput timer you should see less of the behavior
you're describing but there will still be some.

On the other hand I'm wondering what you're trying to prove. Usually a test
exasperates a problem/issue and a "clean" load is not necessary. Or you're
doing a baseline in wich case you're only interested in repeating the same
test again. If you need stats then you can apply the suggestion above. 

Cheers Oliver

On 13/09/11 7:02 AM, "Robin D. Wilson" <rwilson2@gmail.com> wrote:

>Perhaps there is a workaround for this, but I have identified a small 
>flaw in how JMeter is running threads.
>
>Assume for a second that I have a test where I want to run a 100 
>simultaneous users through a procedure. I'd like the test to start and 
>finish running 100 simultaneous threads.
>
>The way JMeter runs threads, it pre-divides the number of iterations 
>among the 100 threads - so if you are going to run 1000 loops, each 
>thread will get 1000 iterations to run through.
>
>Normally, you'd think this was OK... Since each thread will do 1000 
>loops, on 'average' they will work out to do the same amount of work.
>
>But in practice, threads complete their 1000 loops at different 
>intervals, so that by the time the test completes - only a few threads 
>were still running at the very end.
>
>When you are trying to measure performance under load, this skews your 
>result - because the last few iterations occur with almost none of the 
>intended load. This means the last few threads execute very quickly 
>compared to threads operating while under 100 simultaneous requests - 
>so the reported data shows the last few threads having a significant 
>impact on the overall performance of the test.
>
>What I need is a test thread group that runs 'n' simultaneous threads 
>from beginning to end - and runs until 'n' _total_ iterations are 
>complete, instead of just 'n' per thread. Is there a thread group that 
>works like that?
>
>--
>Robin D. Wilson
>Sr. Director of Web Development
>KingsIsle Entertainment, Inc.
>VOICE: 512-777-1861
>www.KingsIsle.com
>
>
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: jmeter-user-unsubscribe@jakarta.apache.org
>For additional commands, e-mail: jmeter-user-help@jakarta.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: jmeter-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: jmeter-user-help@jakarta.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: jmeter-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: jmeter-user-help@jakarta.apache.org