jmeter-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robin D. Wilson" <rwils...@gmail.com>
Subject RE: Average times are confusing me...
Date Wed, 15 Feb 2012 22:58:54 GMT
On JMeter 2.6, setting the settings you recommended fixed the problem:

   - sampleresult.useNanoTime=false
   - sampleresult.nanoThreadSleep=0

But it doesn't do anything for JMeter 2.4. Needless to say, I have a bunch of re-benchmarking
to do...

I can file a ticket - but I am not sure that it will provide much value, since it is clear
that the above settings fix the problem.

--
Robin D. Wilson
Sr. Director of Web Development
KingsIsle Entertainment, Inc.
VOICE: 512-777-1861
www.KingsIsle.com


-----Original Message-----
From: Philippe Mouawad [mailto:philippe.mouawad@gmail.com] 
Sent: Wednesday, February 15, 2012 3:52 PM
To: JMeter Users List
Subject: Re: Average times are confusing me...

Hello,
I think you should submit your Test as a bug just for us to check what
could explain this.
I made many tests on WinXP and never faced this issue so I am not convinced
it's the real explanation although you reproduced it.

Can you also make a test by setting this in jmeter.properties:


   - sampleresult.useNanoTime=false
   - sampleresult.nanoThreadSleep=0


Thanks
Regards
Philippe


On Wed, Feb 15, 2012 at 6:39 PM, Robin D. Wilson <rwilson2@gmail.com> wrote:

> FYI, I have confirmed that my test case only fails on 'WinXP'. When I run
> the exact same test on Win7 (x64), it works fine.
>
> The sad thing is, I've been benchmarking my systems using WinXP for over a
> year ... sigh.
>
> So now I find out that all my benchmark numbers are essentially bogus.
>
> --
> Robin D. Wilson
> Sr. Director of Web Development
> KingsIsle Entertainment, Inc.
> VOICE: 512-777-1861
> www.KingsIsle.com
>
>
> -----Original Message-----
> From: Philippe Mouawad [mailto:philippe.mouawad@gmail.com]
> Sent: Monday, February 13, 2012 2:13 PM
> To: JMeter Users List
> Subject: Re: Average times are confusing me...
>
> Mac OSX 10.6.8.
>
> I suggest you open an issue with your attached test plan (exactly the one
> that has the issue).
>
> Regards
> Philippe
> On Mon, Feb 13, 2012 at 8:28 PM, Robin D. Wilson <rwilson2@gmail.com>
> wrote:
>
> > Just out of curiosity - what type of OS was your JMeter GUI running on?
> > I've been running my tests on Windows XP.
> >
> > --
> > Robin D. Wilson
> > Sr. Director of Web Development
> > KingsIsle Entertainment, Inc.
> > VOICE: 512-777-1861
> > www.KingsIsle.com
> >
> >
> > -----Original Message-----
> > From: Philippe Mouawad [mailto:philippe.mouawad@gmail.com]
> > Sent: Friday, February 10, 2012 7:41 AM
> > To: JMeter Users List
> > Subject: Re: Average times are confusing me...
> >
> > Hello,
> > I have just made the same test with a JSP (doing a Thread.sleep(2000) =>
> > 2s) behind a Tomcat 6 with JMeter 2.6
> >
> > I put 100 threads that loop 100 times in GUI mode (one JMETER):
> > All results shown in Aggregate report are around 2s .
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
>
>
sampler_label,aggregate_report_count,average,aggregate_report_median,aggregate_report_90%_line,aggregate_report_min,aggregate_report
> >
> >
> _max,aggregate_report_error%,aggregate_report_rate,aggregate_report_bandwidth
> > HTTP
> >
> >
> Request,10000,2006,2004,2012,2001,2195,0.0,49.5793194742609,1224.082013722936
> >
> TOTAL,10000,2006,2004,2012,2001,2195,0.0,49.5793194742609,1224.082013722936
> >
> >
> > Summary listener shows this:
> > 2012/02/10 14:36:00 INFO  - jmeter.reporters.Summariser: Generate Summary
> > Results +  2655 in  54,8s =   48,4/s Avg:  2010 Min:  2001 Max:  2195
> > Err:     0 (0,00%)
> > 2012/02/10 14:38:26 INFO  - jmeter.reporters.Summariser: Generate Summary
> > Results +  7345 in 148,9s =   49,3/s Avg:  2005 Min:  2001 Max:  2088
> > Err:     0 (0,00%)
> > 2012/02/10 14:38:26 INFO  - jmeter.reporters.Summariser: Generate Summary
> > Results = 10000 in 201,7s =   49,6/s Avg:  2006 Min:  2001 Max:  2195
> > Err:     0 (0,00%)
> >
> > So as you can see, nothing abnormal .
> >
> > JSP Code:
> >
> > <%@ page  contentType="text/html; charset=UTF-8" %>
> > <html>
> > <body>
> > <%
> > Thread.sleep(2000);
> > %>
> > <!-- GENERATE around 25Ko page -->
> > <%=org.apache.commons.lang.RandomStringUtils.random(25000,
> > "abcdefghijklmnopqrstuvwxyz01234567890")%>
> > <p>Test</p>
> > </body>
> > </html>
> >
> >
> > Regards
> > Philippe
> > http://www.ubik-ingenierie.com
> >
> > On Thu, Feb 9, 2012 at 4:52 PM, sebb <sebbaz@gmail.com> wrote:
> >
> > > On 9 February 2012 15:38, Robin D. Wilson <rwilson2@gmail.com> wrote:
> > > > Thanks sebb for the replies...
> > > >
> > > > Here's the deal, I am running the same test script on JM2.4 and
> JM2.6.
> > I
> > > am running in GUI mode. The test script has 3 thread groups
> > > > - but the first and the last thread group is just a 'timer' I created
> > to
> > > log the total elapsed time of the test (the first and last
> > > > group has 1 thread, and 1 request, and take less than 1 second each
> to
> > > run). The 'real' test is the middle thread group. It has 100
> > > > threads (0 ramp), and runs 100 iterations (10,000 total samples). It
> > > simply does a 'POST' to a URL, with 15
> > > >
> > > > So the 'elapsed time' I referring to in my test is actually the
> > > timestamp taken in the first thread group (in ms since epoch)
> > > > subtracted from the timestamp taken in the 3rd (last) thread group.
> > That
> > > part of my test may only add 2 total seconds to the test,
> > > > so while it may skew my results slightly - it doesn't explain the
> vast
> > > difference in the 'average' sample duration. According to the
> > > > Summary Report docs, the "Average" is supposed to be "the average
> > > elapsed time of a set of samples". But clearly, if the minimum
> > > > time it takes to actually get the page is 2 seconds (due to the
> > built-in
> > > delay in the cgi-script), there is no way I could have an
> > > > 'average' elapsed time of less than 2 seconds, yet I'm showing an
> > > average elapsed time of ~750 ms... (My "Max" elapsed time shows as
> > > > only 1198!). When I request the page in Firefox, it takes ~2104ms
> > (using
> > > a status bar timer), so I think the cgi script is working
> > > > correctly.)
> > > >
> > > > Sebb asked:
> > > >
> > > >>Again, the throughput calculations are based on total test time. Are
> > you
> > > sure the test run times are comparable?
> > > >
> > > > The test run times are automatically calculated by the 1st and 3rd
> > > thread groups. The ~210 seconds total elapsed time is accurate
> > > > based on my external measurement too (e.g., it is close to what I can
> > > observe with my stopwatch).
> > > >
> > > > Both the JM2.4 test and the JM2.6 test are using the exact same
> ".jmx"
> > > test file.
> > > >
> > > >>There's clearly something else going on here.
> > > >
> > > > I don't believe that the Summary Report is accurately calculating
> > > anything except the total number of samples and the Avg. Bytes...
> > >
> > > What makes you say that?
> > > Are the Min and Max really incorrect?
> > > Error %?
> > >
> > > It's easy enough to check the Summary Results if you can provide the
> > > CSV sample result files.
> > >
> > > > The cgi-script I'm using definitely takes 2+ seconds to respond after
> > it
> > > gets the request (I've measured this with Firefox directly,
> > > > and it _never_ gets a response in less than 2 seconds). I even
> changed
> > > the 'sleep' to 9 seconds, and JMeter pauses for that long in
> > > > recording results (e.g., it shows 100 threads run, then waits 9
> > seconds,
> > > shows another 100 threads, etc.), but the numbers just go
> > > > up to '1758' Average, and '2415' Max (which is impossible since it is
> > > taking 9+ seconds to respond to each request!). It takes over
> > > > 15 minutes to complete 10,000 samples (and that seems about right -
> > > 10000 samples/100 threads * 9 seconds each = 900 seconds).
> > > >
> > > > I even went so far as to inject a 2 second sleep in the middle of the
> > > response (e.g., pause 2 seconds -  send part of the response -
> > > > pause 2 more seconds - send the rest), I'm still getting average
> times
> > > of ~1000 ms. (That's with 4 seconds of built-in delays, and 2
> > > > of those seconds are in the middle of the response.) The browser
> shows
> > > this delay properly, but JMeter isn't calculating it
> > > > properly.
> > > >
> > > >>Please recheck the individual sample response times and see how they
> > > compare to the average.
> > > >
> > > > I'm not sure how to do that in JMeter. I can manually hit the page,
> and
> > > it takes about 100ms longer than the built-in delay I have.
> > >
> > > Add a Table View Listener, or just check the CSV sample result files.
> > >
> > > >>If there still appears to be a problem, create a Bugzilla issue and
> > > attach:
> > > >>- JMX test case
> > > >
> > > > I'm trying to simplify the test case to the bare minimum case - so
> the
> > > results will be indisputable. I will also include the
> > > > 'cgi-bin' script that I'm using, so someone else can easily setup the
> > > same test.
> > >
> > > Thanks.
> > >
> > > >
> > > >>- log files for JMeter 2.4 and 2.6
> > > >
> > > > Which log files are these? Is it just the 'jmeter.log' that gets
> > created
> > > in the 'bin' folder when I run the GUI mode, or do you need
> > > > another log file?
> > >
> > > jmeter.log
> > >
> > > >>- CSV result files for 2.4 and 2.6
> > > >
> > > > I can do this.
> > > >
> > > > --
> > > > Robin D. Wilson
> > > > Sr. Director of Web Development
> > > > KingsIsle Entertainment, Inc.
> > > > VOICE: 512-777-1861
> > > > www.KingsIsle.com
> > > >
> > > >
> > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
> > > > For additional commands, e-mail: user-help@jmeter.apache.org
> > > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
> > > For additional commands, e-mail: user-help@jmeter.apache.org
> > >
> > >
> >
> >
> > --
> > Cordialement.
> > Philippe Mouawad.
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
> > For additional commands, e-mail: user-help@jmeter.apache.org
> >
> >
>
>
> --
> Cordialement.
> Philippe Mouawad.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
> For additional commands, e-mail: user-help@jmeter.apache.org
>
>


-- 
Cordialement.
Philippe Mouawad.


---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
For additional commands, e-mail: user-help@jmeter.apache.org


Mime
View raw message