jmeter-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adrian Speteanu <asp.ad...@gmail.com>
Subject Re: Strange 'pause' activity during testing
Date Thu, 15 Dec 2011 11:13:36 GMT
I agree that jmeter rarely performs poorly in non-GUI mode, unless you
allocate to much memory to it.

On Thu, Dec 15, 2011 at 1:12 PM, Adrian Speteanu <asp.adieu@gmail.com>wrote:

> In my experience, "stange pauses" usually points to the application.
>
> Of course, you have to make sure its not you (overlook in the test script,
> jmeter's configuration), or the test client environment (OS, network). BUT
> after all that is exhausted is usually the application or one of the layers
> of the system under test (like the database, usually more trivial stuff
> like a small connection pool to the database, or GC of the application
> under test).
>
> If you suspect its GC, divide the load you generate with the script in 3,
> and instead of testing with one jmeter instance, use 3 (if you can test
> like this on 3 different machines - the better). Also allocate less memory
> to jmeter now. What's the probability that all instances get paused by GC
> at the same time? Also, if you allocate less memory, stop-the-world GC
> takes less time. If you still have response times pauses in the same
> intervals - it has to be the system under test. Note:
>
> On Thu, Dec 15, 2011 at 3:51 AM, Robin D. Wilson <rwilson2@gmail.com>wrote:
>
>> Yes it is. I can turn it off if you think that matters.
>>
>> --
>> Robin D. Wilson
>>
>>
>>
>> On Dec 14, 2011, at 6:42 PM, basim@baassiri.ca wrote:
>>
>> > I'm curious to know if the system under test is https
>> >
>> >
>> > -----Original Message-----
>> > From: sebb <sebbaz@gmail.com>
>> > Date: Thu, 15 Dec 2011 00:35:39
>> > To: JMeter Users List<user@jmeter.apache.org>
>> > Reply-To: "JMeter Users List" <user@jmeter.apache.org>
>> > Subject: Re: Strange 'pause' activity during testing
>> >
>> > On 15 December 2011 00:22, Robin D. Wilson <rwilson2@gmail.com> wrote:
>> >> I can use non-GUI mode for testing out the problem, but with the exact
>> same
>> >> test - just switching the 'Retrieve All Embedded...' disabled, the
>> problem
>> >> goes away.
>> >>
>> >> The only listeners I have enabled are the summary listener - the
>> PerfMon
>> >> listeners (1 for each server) - and a 'tree' listener, but it only
>> reports
>> >> 'errors'.
>> >>
>> >> I can run the test with 'Retrieve All Embedded...' enabled, I see the
>> >> problem every time.
>> >
>> > Maybe then there is a problem with the scanning of the HTML to extract
>> > the embedded resources, or maybe one of the embedded resources is a
>> > tar-pit.
>> >
>> > Do all the expected embedded resources get downloaded?
>> >
>> > Are there any unusually long elapsed times for embedded resources?
>> >
>> > Or large gaps between the parent sample download completion and the
>> > start of the first embedded resource?
>> >
>> > That would suggest the page took a while to parse.
>> >
>> > You'll need to select the optiion to "save subresults" in order to see
>> > the embedded samples.
>> >
>> >> If I just uncheck the box in the Request Defaults and re-run the test,
>> it
>> >> never happens... So are you suggesting that I've hit some arbitrary
>> limit of
>> >> the GUI mode?
>> >
>> > No, just that GUI mode is inherently more resource intensive.
>> >
>> > BTW, you wrote you were running JMeter 2.4.1 - that does not exist, so
>> > perhaps you meant the current version, 2.5.1?
>> >
>> >
>> >> What I'm wondering is why:
>> >>
>> >> 1) It is so consistent - the 'stop' happens at nearly the exact same
>> point
>> >> in every test run, and for nearly the exact same duration - and it
>> happens
>> >> multiple times during the test run, in the same pattern.
>> >>
>> >> 2) I can increase the load 10X (even more if I want), but without the
>> >> 'Retrieve All Embedded...' enabled, and it doesn’t happen.
>> >>
>> >> If it were resource related on the JMeter box, I would guess that
>> changing
>> >> the JVM memory config would show significant changes in behavior (like
>> it
>> >> would take longer to encounter the first 'stop'). If it were resource
>> >> related on the servers, I would expect to see some sort of contention
>> on the
>> >> servers.
>> >>
>> >> Tomorrow I will try the thread dump stuff - to see if there is anything
>> >> obvious in that output.
>> >>
>> >> From an "external" view - it looks like JMeter has some sort of 'pause'
>> >> built into it. Since all machines involved are nearly completely
>> 'idle' when
>> >> the stop happens - it seems unlikely that I've run out of resources.
>> The
>> >> other thought is that there is a deadlock somewhere - and something
>> has to
>> >> timeout before it can free up the deadlock and start again. But if
>> that were
>> >> the case, I would expect it to happen at slightly different intervals
>> in
>> >> each test run - and not repeat in the same consistent pattern during
>> the
>> >> test run.
>> >>
>> >> Some thoughts on what I think I've ruled out:
>> >>
>> >> 1) If it were the network 'backing off' I would expect the PerfMon
>> listener
>> >> to stop updating during the 'stop' - but it trundles happily along.
>> >> 2) If it were JVM garbage collections - I would have expected to see
>> them
>> >> running (or at least a 'stop-the-world' gc right before/after the
>> 'stop'
>> >> period) during the 'stop' period - but there are no gc events at all
>> during
>> >> the 'stop' period.
>> >> 3) If it were WinXP resources, I would have expected the PerfMon
>> listener to
>> >> show some activity during the 'stop' period on the WinXP (JMeter
>> client) box
>> >> - nothing like that appears.
>> >>
>> >> I'll look at the thread dumps tomorrow and get back to you guys.
>> Thanks for
>> >> helping me ponder this.
>> >>
>> >> --
>> >> Robin D. Wilson
>> >> Sr. Director of Web Development
>> >> KingsIsle Entertainment, Inc.
>> >> VOICE: 512-777-1861
>> >> www.KingsIsle.com
>> >>
>> >>
>> >> -----Original Message-----
>> >> From: sebb [mailto:sebbaz@gmail.com]
>> >> Sent: Wednesday, December 14, 2011 5:27 PM
>> >> To: JMeter Users List
>> >> Subject: Re: Strange 'pause' activity during testing
>> >>
>> >> On 14 December 2011 22:20, Deepak Shetty <shettyd@gmail.com> wrote:
>> >>> I think the original post mentioned that GC was not running.
>> >>>
>> >>> can you take a thread dump when there is a pause and see what the
>> >>> threads are waiting on?
>> >>>
>> >>
>> >> I would also try running JMeter in non-GUI mode; GUI mode is very
>> expensive
>> >> in resources.
>> >>
>> >> Ensure all non-essential Listeners are disabled, see:
>> >>
>> >> http://jmeter.apache.org/usermanual/best-practices.html#lean_mean
>> >>
>> >>>
>> >>> On Wed, Dec 14, 2011 at 1:56 PM, Kirk <kirk.pepperdine@gmail.com>
>> wrote:
>> >>>
>> >>>> might be GC.. me be that JMeter's threads are all hung up in your
>> >>>> server doing stuff.
>> >>>>
>> >>>> Regards,
>> >>>> Kirk
>> >>>> On Dec 14, 2011, at 10:35 PM, Robin D. Wilson wrote:
>> >>>>
>> >>>>> I have a marginally complicated test case that performs a
>> >> 'registration'
>> >>>> on
>> >>>>> my site. It gets the home page, POSTS a form, gets the response,
>> >>>>> POSTS another form, gets that response, and then completes.
The
>> >>>>> test runs fine with 100 threads, and 30000 iterations - IF I
don't
>> >>>>> "Retrieve All
>> >>>> Embedded
>> >>>>> Resources from HTML Files". In this mode, I am really testing
the
>> >>>> throughput
>> >>>>> of my 'tomcat' application, not the other elements of my system.
>> >>>>> (I'm assuming that the other elements are being retrieved from
our
>> >>>>> Content
>> >>>> Data
>> >>>>> Network instead of our main system in this case.)
>> >>>>>
>> >>>>> If I enable the "Retrieve All Embedded Resources from HTML Files"
>> >>>>> flag,
>> >>>> and
>> >>>>> tune the test down to 10 threads with 3000 total iterations,
I
>> >>>>> notice a
>> >>>> very
>> >>>>> strange behavior. The test runs along at a pretty good clip
for the
>> >>>>> first
>> >>>>> ~600 iterations (about 1 min, 25 seconds into the run), and
then it
>> >>>>> just stops making requests for about 35 seconds. Then, it picks
>> >>>>> back up again
>> >>>> and
>> >>>>> runs for another 1 m 25s, and then stops again for 35 seconds...
>> (NOTE:
>> >>>> with
>> >>>>> the 10 threads, 3000 total iterations - but with "Retrieve All
>> >>>> Embedded..."
>> >>>>> disabled - I don't see the 'stop' behavior either - so it isn't
>> >>>>> caused by tuning it down...)
>> >>>>>
>> >>>>> I recently added the "Perfmon Metrics Collector" to the test
>> >>>>> script, so I could see if one of the servers was maxed out -
but it
>> >>>>> looks like all the servers are idle during the 'stop' period.
>> >>>>> Likewise, I added the Perfmon
>> >>>> for
>> >>>>> the "localhost" (running the JMeter test) to see if it was swamped
>> >>>>> - but
>> >>>> it
>> >>>>> too is idle during the 'stop'. I swapped out our network switch
>> >>>>> (the test environment is on an isolated network switch) with
a
>> >>>>> _much_ higher
>> >>>> capacity
>> >>>>> switch - in case there was a network issue, still no change.
>> >>>>>
>> >>>>> I'm running out of ideas for things to check - so I thought
I'd ask
>> >>>>> you
>> >>>> guys
>> >>>>> if you have any suggestions of things I should look at.
>> >>>>>
>> >>>>> My system consists of:
>> >>>>>
>> >>>>>       WinXP - running JMeter 2.4.1 - driving the test script
in GUI
>> >>>>> mode
>> >>>>>       Server 1 - running Red Hat Linux, with "Apache (2.2.21)"
as
>> >>>>> the web server - using AJP Proxy to Server 2
>> >>>>>       Server 2 - running Red Hat Linux, with Tomcat 7.0.21 as
the
>> >>>>> App Server - connecting through Hibernate to Server 3
>> >>>>>       Server 3 - running Red Hat Linux with MySQL 5.x as the
DB
>> >>>>> Server
>> >>>>>
>> >>>>> All 4 machines are running on a private switched network (32Gbs
>> >>>> backplane).
>> >>>>>
>> >>>>> The requests are downloading about 3MB total (per thread per
>> >>>>> iteration)
>> >>>> over
>> >>>>> 4 main URL requests, and 30+ 'Retrieve All Embedded' requests.
>> >>>>>
>> >>>>> At first I thought it was the network - but the new switch seemed
>> >>>>> to deny that thought (the old switch had a much slower backplane).
>> >>>>> Also, I'm
>> >>>> having
>> >>>>> no trouble collecting the PerfMon data during the 'stop' period
-
>> >>>>> so the network is still functioning just fine...
>> >>>>> Then I thought it might be garbage collection on the tomcat
- but I
>> >>>> watched
>> >>>>> the gc.log - and it doesn't do any GCs during the 'stop' period.
>> >>>>> Then I thought it might be the garbage collection on the JMeter
>> >>>>> side, so
>> >>>> I
>> >>>>> started the JMeter.bat from a 'cmd' prompt with gc logging enabled
>> >>>>> - it doesn't do any GCs during the 'stop' period either.
>> >>>>>
>> >>>>> The apache, tomcat, and DB are all 'idle' (no CPU to speak of,
no
>> >>>>> network I/O, no disk I/O, etc.) during the 'stop' period.
>> >>>>>
>> >>>>> The JMeter box (WinXP) is doing very little during that time
too (I
>> >>>>> attribute the little bit of activity to displaying the PerfMon
>> >>>>> graphs,
>> >>>> and
>> >>>>> Remote Desktop display to my desktop computer)...
>> >>>>>
>> >>>>> --
>> >>>>> Robin D. Wilson
>> >>>>> Sr. Director of Web Development
>> >>>>> KingsIsle Entertainment, Inc.
>> >>>>> VOICE: 512-777-1861
>> >>>>> www.KingsIsle.com
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> -------------------------------------------------------------------
>> >>>>> -- To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
>> >>>>> For additional commands, e-mail: user-help@jmeter.apache.org
>> >>>>>
>> >>>>
>> >>>>
>> >>>> ---------------------------------------------------------------------
>> >>>> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
>> >>>> For additional commands, e-mail: user-help@jmeter.apache.org
>> >>>>
>> >>>>
>> >>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
>> >> For additional commands, e-mail: user-help@jmeter.apache.org
>> >>
>> >>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
>> >> For additional commands, e-mail: user-help@jmeter.apache.org
>> >>
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
>> > For additional commands, e-mail: user-help@jmeter.apache.org
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
>> For additional commands, e-mail: user-help@jmeter.apache.org
>>
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message