jmeter-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robin D. Wilson" <rwils...@gmail.com>
Subject RE: Average times are confusing me...
Date Wed, 15 Feb 2012 17:39:00 GMT
FYI, I have confirmed that my test case only fails on 'WinXP'. When I run the exact same test
on Win7 (x64), it works fine.

The sad thing is, I've been benchmarking my systems using WinXP for over a year ... sigh.


So now I find out that all my benchmark numbers are essentially bogus.

--
Robin D. Wilson
Sr. Director of Web Development
KingsIsle Entertainment, Inc.
VOICE: 512-777-1861
www.KingsIsle.com


-----Original Message-----
From: Philippe Mouawad [mailto:philippe.mouawad@gmail.com] 
Sent: Monday, February 13, 2012 2:13 PM
To: JMeter Users List
Subject: Re: Average times are confusing me...

Mac OSX 10.6.8.

I suggest you open an issue with your attached test plan (exactly the one
that has the issue).

Regards
Philippe
On Mon, Feb 13, 2012 at 8:28 PM, Robin D. Wilson <rwilson2@gmail.com> wrote:

> Just out of curiosity - what type of OS was your JMeter GUI running on?
> I've been running my tests on Windows XP.
>
> --
> Robin D. Wilson
> Sr. Director of Web Development
> KingsIsle Entertainment, Inc.
> VOICE: 512-777-1861
> www.KingsIsle.com
>
>
> -----Original Message-----
> From: Philippe Mouawad [mailto:philippe.mouawad@gmail.com]
> Sent: Friday, February 10, 2012 7:41 AM
> To: JMeter Users List
> Subject: Re: Average times are confusing me...
>
> Hello,
> I have just made the same test with a JSP (doing a Thread.sleep(2000) =>
> 2s) behind a Tomcat 6 with JMeter 2.6
>
> I put 100 threads that loop 100 times in GUI mode (one JMETER):
> All results shown in Aggregate report are around 2s .
>
>
>
>
>
>
>
>
>
>
>
>
sampler_label,aggregate_report_count,average,aggregate_report_median,aggregate_report_90%_line,aggregate_report_min,aggregate_report
>
> _max,aggregate_report_error%,aggregate_report_rate,aggregate_report_bandwidth
> HTTP
>
> Request,10000,2006,2004,2012,2001,2195,0.0,49.5793194742609,1224.082013722936
> TOTAL,10000,2006,2004,2012,2001,2195,0.0,49.5793194742609,1224.082013722936
>
>
> Summary listener shows this:
> 2012/02/10 14:36:00 INFO  - jmeter.reporters.Summariser: Generate Summary
> Results +  2655 in  54,8s =   48,4/s Avg:  2010 Min:  2001 Max:  2195
> Err:     0 (0,00%)
> 2012/02/10 14:38:26 INFO  - jmeter.reporters.Summariser: Generate Summary
> Results +  7345 in 148,9s =   49,3/s Avg:  2005 Min:  2001 Max:  2088
> Err:     0 (0,00%)
> 2012/02/10 14:38:26 INFO  - jmeter.reporters.Summariser: Generate Summary
> Results = 10000 in 201,7s =   49,6/s Avg:  2006 Min:  2001 Max:  2195
> Err:     0 (0,00%)
>
> So as you can see, nothing abnormal .
>
> JSP Code:
>
> <%@ page  contentType="text/html; charset=UTF-8" %>
> <html>
> <body>
> <%
> Thread.sleep(2000);
> %>
> <!-- GENERATE around 25Ko page -->
> <%=org.apache.commons.lang.RandomStringUtils.random(25000,
> "abcdefghijklmnopqrstuvwxyz01234567890")%>
> <p>Test</p>
> </body>
> </html>
>
>
> Regards
> Philippe
> http://www.ubik-ingenierie.com
>
> On Thu, Feb 9, 2012 at 4:52 PM, sebb <sebbaz@gmail.com> wrote:
>
> > On 9 February 2012 15:38, Robin D. Wilson <rwilson2@gmail.com> wrote:
> > > Thanks sebb for the replies...
> > >
> > > Here's the deal, I am running the same test script on JM2.4 and JM2.6.
> I
> > am running in GUI mode. The test script has 3 thread groups
> > > - but the first and the last thread group is just a 'timer' I created
> to
> > log the total elapsed time of the test (the first and last
> > > group has 1 thread, and 1 request, and take less than 1 second each to
> > run). The 'real' test is the middle thread group. It has 100
> > > threads (0 ramp), and runs 100 iterations (10,000 total samples). It
> > simply does a 'POST' to a URL, with 15
> > >
> > > So the 'elapsed time' I referring to in my test is actually the
> > timestamp taken in the first thread group (in ms since epoch)
> > > subtracted from the timestamp taken in the 3rd (last) thread group.
> That
> > part of my test may only add 2 total seconds to the test,
> > > so while it may skew my results slightly - it doesn't explain the vast
> > difference in the 'average' sample duration. According to the
> > > Summary Report docs, the "Average" is supposed to be "the average
> > elapsed time of a set of samples". But clearly, if the minimum
> > > time it takes to actually get the page is 2 seconds (due to the
> built-in
> > delay in the cgi-script), there is no way I could have an
> > > 'average' elapsed time of less than 2 seconds, yet I'm showing an
> > average elapsed time of ~750 ms... (My "Max" elapsed time shows as
> > > only 1198!). When I request the page in Firefox, it takes ~2104ms
> (using
> > a status bar timer), so I think the cgi script is working
> > > correctly.)
> > >
> > > Sebb asked:
> > >
> > >>Again, the throughput calculations are based on total test time. Are
> you
> > sure the test run times are comparable?
> > >
> > > The test run times are automatically calculated by the 1st and 3rd
> > thread groups. The ~210 seconds total elapsed time is accurate
> > > based on my external measurement too (e.g., it is close to what I can
> > observe with my stopwatch).
> > >
> > > Both the JM2.4 test and the JM2.6 test are using the exact same ".jmx"
> > test file.
> > >
> > >>There's clearly something else going on here.
> > >
> > > I don't believe that the Summary Report is accurately calculating
> > anything except the total number of samples and the Avg. Bytes...
> >
> > What makes you say that?
> > Are the Min and Max really incorrect?
> > Error %?
> >
> > It's easy enough to check the Summary Results if you can provide the
> > CSV sample result files.
> >
> > > The cgi-script I'm using definitely takes 2+ seconds to respond after
> it
> > gets the request (I've measured this with Firefox directly,
> > > and it _never_ gets a response in less than 2 seconds). I even changed
> > the 'sleep' to 9 seconds, and JMeter pauses for that long in
> > > recording results (e.g., it shows 100 threads run, then waits 9
> seconds,
> > shows another 100 threads, etc.), but the numbers just go
> > > up to '1758' Average, and '2415' Max (which is impossible since it is
> > taking 9+ seconds to respond to each request!). It takes over
> > > 15 minutes to complete 10,000 samples (and that seems about right -
> > 10000 samples/100 threads * 9 seconds each = 900 seconds).
> > >
> > > I even went so far as to inject a 2 second sleep in the middle of the
> > response (e.g., pause 2 seconds -  send part of the response -
> > > pause 2 more seconds - send the rest), I'm still getting average times
> > of ~1000 ms. (That's with 4 seconds of built-in delays, and 2
> > > of those seconds are in the middle of the response.) The browser shows
> > this delay properly, but JMeter isn't calculating it
> > > properly.
> > >
> > >>Please recheck the individual sample response times and see how they
> > compare to the average.
> > >
> > > I'm not sure how to do that in JMeter. I can manually hit the page, and
> > it takes about 100ms longer than the built-in delay I have.
> >
> > Add a Table View Listener, or just check the CSV sample result files.
> >
> > >>If there still appears to be a problem, create a Bugzilla issue and
> > attach:
> > >>- JMX test case
> > >
> > > I'm trying to simplify the test case to the bare minimum case - so the
> > results will be indisputable. I will also include the
> > > 'cgi-bin' script that I'm using, so someone else can easily setup the
> > same test.
> >
> > Thanks.
> >
> > >
> > >>- log files for JMeter 2.4 and 2.6
> > >
> > > Which log files are these? Is it just the 'jmeter.log' that gets
> created
> > in the 'bin' folder when I run the GUI mode, or do you need
> > > another log file?
> >
> > jmeter.log
> >
> > >>- CSV result files for 2.4 and 2.6
> > >
> > > I can do this.
> > >
> > > --
> > > Robin D. Wilson
> > > Sr. Director of Web Development
> > > KingsIsle Entertainment, Inc.
> > > VOICE: 512-777-1861
> > > www.KingsIsle.com
> > >
> > >
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
> > > For additional commands, e-mail: user-help@jmeter.apache.org
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
> > For additional commands, e-mail: user-help@jmeter.apache.org
> >
> >
>
>
> --
> Cordialement.
> Philippe Mouawad.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
> For additional commands, e-mail: user-help@jmeter.apache.org
>
>


-- 
Cordialement.
Philippe Mouawad.


---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
For additional commands, e-mail: user-help@jmeter.apache.org


Mime
View raw message