jmeter-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tuukka Mustonen <tuukka.musto...@gmail.com>
Subject Re: High CPU utilization in JMeter 3.x with HttpClient 4 leads to freeze
Date Fri, 03 Feb 2017 13:40:37 GMT
Sorry for the delay, got too busy yesterday.

I have been running more tests now. I tried to a few variations to pinpoint
the issue, but first:

@Philippe:

0) Response headers are like:

Response headers:
HTTP/1.1 200 OK
Date: Fri, 03 Feb 2017 06:47:00 GMT
Content-Type: application/json
Content-Length: 374
Connection: keep-alive
Server: nginx/1.6.2
ETag: 37926:1-37927:1
Cache-Control: no-cache, no-store
Pragma: no-cache
Expires: 0
X-Transaction: 0000-326srxia31tss3tc

1) Tried your test plan. Unfortunately, it does not make difference.

2) I ran Wireshark during a couple of tests. You can find a few screenshots
in a later link. Note that my whole network kinda crashes when JMeter 3
gets stuck (ping gets "General failure" and some active connections crash,
note that this seems a bit random).

3) I gave it a few minutes to recover between the runs.

@Felix:

I ran all tests with loggerpanel.usejsyntaxtext=false.

No, the amount of seems roughly the same.



Then, about pinpointing the issue:

I managed to reproduce this by installing nginx on a network computer (sudo
apt-get install nginx on debian 8). I did not configure anything and I
simply bombarded the default nginx welcome page.

I have attached Heap Dump, Thread Dumps and some other information in these
links:

JAVA impl (that doesn't crash):
https://drive.google.com/drive/folders/0B1uBdVuLED5NSFJvc2VSdE13a0k?usp=sharing

HttpCilent 4 impl (that crashes):
https://drive.google.com/drive/folders/0B1uBdVuLED5NUGc1WWFYYkRncTQ?usp=sharing

I used Philippe's test plan and even simplified it a bit, the JMX file is
in the latter link. Note that the errors occur (in my case) during the
ramp-up so I also shortened to run just 30s after reaching 500 users.

With JAVA impl, I ran heap dump at ~450 users or so, and thread dumps after
that.

With HttpClient 4 impl, I ran heap dump immediately after seeing the first
error (which was at ~430 users or so) and thread dumps right after that
(the first two dumps I think took a bit longer than normally to create).
JMeter goes unresponsive a second or two after seeing the first error, so I
just barely had time to click to take the dump.

As said, there are some screenshots from Wireshark trace for HttpClient 4
run. I am not network expert, so I don't dare to judge them, but I included
what I saw interesting. I think I can share some parts (privately) should
you want to study the trace in detail.

Finally, when I disabled keep-alive from nginx configuration, the problem
was gone (as earlier). So, the problem is somehow related to keep-alive.

Note that in all scenarios CPU goes to 100% quite immediately (also on 2.13
now, probably because of high throughput, I assume my workstation is the
bottleneck here) but without keep-alive or with JAVA impl, things work and
with keep-alive enabled and HttpClient 4, things break apart. As before, on
JMeter 2.13 everything works.

Tuukka



On Fri, Feb 3, 2017 at 7:57 AM, Felix Schumacher <
felix.schumacher@internetallee.de> wrote:

>
>
> Am 2. Februar 2017 06:52:32 MEZ schrieb Tuukka Mustonen <
> tuukka.mustonen@gmail.com>:
> >@Philippe:
> >
> >0. Alright, I will.
> >1. Will do.
> >2. Sure.
> >3. To let the network connection state restore? Ok, will do.
> >
> >I am having some other stuff I need to attend today, but I will get
> >back to
> >this in the afternoon at latest.
>
> Could you also try to set loggerpanel.usejsyntaxtext to false in your
> settings?
>
> Are there more log entries, when you use httpclient 4 instead of Java?
>
> Vladimir has found signs for a event congestion in the awt event queue,
> that show in your threaddumps.
>
> Regards,
>  Felix
>
> >
> >Tuukka
> >
> >
> >On Wed, Feb 1, 2017 at 11:54 PM, Philippe Mouawad <
> >philippe.mouawad@gmail.com> wrote:
> >
> >> Hello,
> >> I've created this bugzilla for investigation and to share with you
> >the plan
> >> I have used to run my own tests:
> >> - https://bz.apache.org/bugzilla/show_bug.cgi?id=60676
> >>
> >> I use a Tomcat 8.5.11 with a localhost and a certificate generated
> >and auto
> >> signed.
> >>
> >> So here is what I get on a Mac Book Pro Mid 2015 OSX El Capitan 2.5
> >Ghz
> >> Intel Core i7
> >> Java 8 u121
> >> -Xmx512m -Xms512m
> >> Test in GUI mode (ANTI-PATTERN)
> >>
> >> 2.13:
> >> - CPU is at 100%
> >> - No error
> >> - No leak
> >>
> >> 3.2 nightly with default settings :
> >> - CPU is at 100%
> >> - No error
> >> - No leak
> >>
> >> 3.2 nightly with this in user.properties :
> >> httpclient4.time_to_live=0
> >> httpclient4.validate_after_inactivity=0
> >>
> >> - CPU is at 100%
> >> - No error
> >> - No leak
> >>
> >> I attached the Heap Graphs to bugzilla and the plan.
> >>
> >> So for now I don't  reproduce your issue but I am not in the same
> >> configuration as I am not hitting the same website.
> >>
> >> So
> >> 0/ Could you provide by running only 5 requests, the response header
> >that
> >> your website returns ?
> >> 1/ Could you try my test plan by replacing with your target host and
> >see if
> >> behaviour is the same ?
> >> 2/ Ensure nothing is running (No Spotify :-) ) except your JMeter and
> >> monitor network as per Felix note
> >> 3/ Run 1 test after the other but ensure you wait a bit between each
> >one
> >>
> >>
> >> Thanks
> >>
> >>
> >> On Wed, Feb 1, 2017 at 10:00 PM, Felix Schumacher <
> >> felix.schumacher@internetallee.de> wrote:
> >>
> >> >
> >> >
> >> > Am 1. Februar 2017 12:57:01 MEZ schrieb Tuukka Mustonen <
> >> > tuukka.mustonen@gmail.com>:
> >> > >Ok I re-ran the test.
> >> > >
> >> > >Took Heap Dump just before JMeter froze.
> >> > >
> >> > >Took Thread Dumps right after JMeter froze.
> >> > >
> >> >
> >>https://drive.google.com/drive/folders/0B1uBdVuLED5NNFNqallGam13MTA
> >> >
> >> > That folder seems to be empty now.
> >> >
> >> > Are you running the tests with a security manager enabled?
> >> >
> >> > Can you share the test or at least the used requests?
> >> >
> >> > Are you using a proxy and if so, how is it configured?
> >> >
> >> > Is jmeter running locally, or over something like rdp?
> >> >
> >> > Can you look at the used network connections like netstat -etapn
> >would do
> >> > on Linux?
> >> >
> >> > Regards,
> >> >  Felix
> >> >
> >> > >
> >> > >...
> >> > >
> >> > >For the fun of it, I also tried upgrading to the latest HttpClient
> >> > >4.5.3
> >> > >version. No effect.
> >> > >
> >> > >I also noticed that if I ping (google) during the perf testing,
> >Windows
> >> > >ping reports "General failure." when JMeter breaks. After killing
> >it,
> >> > >connectivity restores and there are replies from the server again.
> >It
> >> > >seems
> >> > >like JMeter/HttpClient kills my whole network connection...
> >> > >
> >> > >Tuukka
> >> > >
> >> > >
> >> > >On Wed, Feb 1, 2017 at 11:44 AM, Philippe Mouawad <
> >> > >philippe.mouawad@gmail.com> wrote:
> >> > >
> >> > >> On Wed, Feb 1, 2017 at 9:55 AM, Tuukka Mustonen
> >> > ><tuukka.mustonen@gmail.com
> >> > >> >
> >> > >> wrote:
> >> > >>
> >> > >> > Hi, I'm running solely on Win 10 now. In re-attempts I think
I
> >saw
> >> > >> slightly
> >> > >> > different behavior in my Linux guest. I'll write more about
> >that
> >> > >later,
> >> > >> but
> >> > >> > for simplicity I'm testing only on Windows host now:
> >> > >> >
> >> > >> > Thread dumps are downloadable at:
> >> > >> >
> >https://drive.google.com/drive/folders/0B1uBdVuLED5NS1RFbEFuSS1iejA
> >> > >> >
> >> > >> > (Took them with 7-8s interval I think, hope it's ok).
> >> > >> >
> >> > >> > I took the thread dumps once JMeter started filling in errors,
> >> > >which is
> >> > >> > somewhat after my CPU usage hit 100% (and music playback
> >started to
> >> > >> distort
> >> > >> > etc.).
> >> > >> >
> >> > >> > I also created the head dump, note that I took it after
> >running the
> >> > >test
> >> > >> > and after restarting JMeter, as JMeter froze during the test
> >and I
> >> > >had to
> >> > >> > kill it. DId you mean I should take it *during* the test?
> >> > >>
> >> > >>
> >> > >> YES.
> >> > >>
> >> > >>
> >> > >> > Anyway, before
> >> > >> > sharing it I want to make sure it can't contain anything
> >> > >confidential?
> >> > >> > (parts of my test plan, passwords, environment variables,
> >> > >anything...)
> >> > >> >
> >> > >> > Tuukka
> >> > >> >
> >> > >> >
> >> > >> >
> >> > >> > On Wed, Feb 1, 2017 at 10:10 AM, UBIK LOAD PACK Support <
> >> > >> > support@ubikloadpack.com> wrote:
> >> > >> >
> >> > >> > > Hello,
> >> > >> > > To investigate the problem:
> >> > >> > > 1/ to create  thread dumps, install a JDK of same version
as
> >the
> >> > >one
> >> > >> > > running your JMeter.
> >> > >> > >
> >> > >> > > Run until you get the problem, then call using same
user as
> >the
> >> > >one
> >> > >> > running
> >> > >> > > JMeter 3 times at 5s interval:
> >> > >> > >  <jdk_home>/bin/jstack -l <pid of JMeter>
>> output_file.txt
> >> > >> > >
> >> > >> > > It will also help if you provide the structure of your
Test
> >Plan.
> >> > >> > >
> >> > >> > > 2/ It would also help if you can provide a Heap Dump.
With
> >JMeter
> >> > >GUI,
> >> > >> > use
> >> > >> > > menu Help > Create a Heap Dump
> >> > >> > > This will create a Heap Dump (rather big file) , it
would
> >help a
> >> > >lot if
> >> > >> > you
> >> > >> > > can make it available.
> >> > >> > >
> >> > >> > > Regards
> >> > >> > >
> >> > >> > >
> >> > >> > >
> >> > >> > >
> >> > >> > >
> >> > >> > >
> >> > >> > >
> >> > >> > > On Wed, Feb 1, 2017 at 9:03 AM, Tuukka Mustonen <
> >> > >> > tuukka.mustonen@gmail.com
> >> > >> > > >
> >> > >> > > wrote:
> >> > >> > >
> >> > >> > > > @Philippe:
> >> > >> > > >
> >> > >> > > > Yeah, I am normally running in non-GUI mode, but
to
> >> > >> > > > study/inspect/understand this problem I am running
in GUI
> >mode.
> >> > >> JMeter
> >> > >> > > 2.13
> >> > >> > > > works just fine in GUI mode for these tests so
JMeter 3.x
> >> > >should too.
> >> > >> > > >
> >> > >> > > > I'll give the nightly build a try.
> >> > >> > > >
> >> > >> > > > Tuukka
> >> > >> > > >
> >> > >> > > >
> >> > >> > > > On Wed, Feb 1, 2017 at 9:59 AM, Philippe Mouawad
<
> >> > >> > > > philippe.mouawad@gmail.com
> >> > >> > > > > wrote:
> >> > >> > > >
> >> > >> > > > > On Wednesday, February 1, 2017, Tuukka Mustonen
<
> >> > >> > > > tuukka.mustonen@gmail.com
> >> > >> > > > > >
> >> > >> > > > > wrote:
> >> > >> > > > >
> >> > >> > > > > > @Philippe:
> >> > >> > > > > >
> >> > >> > > > > > 1. Yeah, I compared them with the same
plan, same
> >machine.
> >> > >Should
> >> > >> > be
> >> > >> > > > > > everything equal.
> >> > >> > > > > > 2. I tried both the defaults in jmeter.bat
(-Xms512m
> >> > >-Xmx512m)
> >> > >> and
> >> > >> > as
> >> > >> > > > > > slightly increased (-Xms512m -Xmx1024m).
I didn't
> >tweak
> >> > >them much
> >> > >> > > > > because I
> >> > >> > > > > > don't believe the issue lies there.
> >> > >> > > > > > 3. No, I do just simple HTTP GETs (against
RESTy API).
> >> > >> > > > > > 4. I am not familiar with that but I
can try. If you
> >have
> >> > >> > > instructions
> >> > >> > > > to
> >> > >> > > > > > point to, please do.
> >> > >> > > > >
> >> > >> > > > >
> >> > >> > > > >
> >> > >> > > > >
> >> > >> > > > > > 5. I am running with GUI.
> >> > >> > > > >
> >> > >> > > > >
> >> > >> > > > > This is a known anti-pattern .
> >> > >> > > > > switch to non gui mode if you want correct
results.
> >> > >> > > > > http://jmeter.apache.org/usermanual/best-practices.html
> >> > >> > > > >
> >> > >> > > > > http://www.ubik-ingenierie.com/blog/jmeter_performance_
> >> > >> tuning_tips/
> >> > >> > > > >
> >> > >> > > > >
> >> > >> > > > > > About file sending:
> >> > >> > > > > >
> >> > >> > > > > > 6. Nope. I'm just doing simple GETs.
> >> > >> > > > >
> >> > >> > > > >  Could you give nightly build a try ?
> >> > >> > > > > http://jmeter.apache.org/nightly.html
> >> > >> > > > >
> >> > >> > > > >
> >> > >> > > > > > Tuukka
> >> > >> > > > > >
> >> > >> > > > > >
> >> > >> > > > > > On Wed, Feb 1, 2017 at 9:43 AM, Philippe
Mouawad <
> >> > >> > > > > > philippe.mouawad@gmail.com <javascript:;>
> >> > >> > > > > > > wrote:
> >> > >> > > > > >
> >> > >> > > > > > > Also are you sending files in post
body ?
> >> > >> > > > > > >
> >> > >> > > > > > > Thank you
> >> > >> > > > > > >
> >> > >> > > > > > > On Wednesday, February 1, 2017,
Philippe Mouawad <
> >> > >> > > > > > > philippe.mouawad@gmail.com <javascript:;>>
> >> > >> > > > > > > wrote:
> >> > >> > > > > > >
> >> > >> > > > > > > > Hello,
> >> > >> > > > > > > > Few questions, please try to
answer them all:
> >> > >> > > > > > > >
> >> > >> > > > > > > > 1/ Did you compare 2.13 and
3.1 with same
> >plan/sape
> >> > >> machine...
> >> > >> > > > > > everything
> >> > >> > > > > > > > equal ?
> >> > >> > > > > > > > 2/ What are your Xmx settings
> >> > >> > > > > > > > 3/ does your plan download
embedded resources ?
> >> > >> > > > > > > > 4/ when issue happens can you
make 3 thread dumps
> >at 5
> >> > >> seconds
> >> > >> > > > > interval
> >> > >> > > > > > > > and send file ? (use jstack)
> >> > >> > > > > > > > 5/ are you running gui or non
gui ?
> >> > >> > > > > > > >
> >> > >> > > > > > > > Thanks
> >> > >> > > > > > > > Regards
> >> > >> > > > > > > >
> >> > >> > > > > > > > On Wednesday, February 1, 2017,
Tuukka Mustonen <
> >> > >> > > > > > > tuukka.mustonen@gmail.com <javascript:;>
> >> > >> > > > > > > >
> ><javascript:_e(%7B%7D,'cvml','tuukka.mustonen@gmail.com
> >> > >> > > > > <javascript:;>');>>
> >> > >> > > > > > wrote:
> >> > >> > > > > > > >
> >> > >> > > > > > > >> Hi,
> >> > >> > > > > > > >>
> >> > >> > > > > > > >> After upgrading to JMeter
3.0 or 3.1 I experience
> >high
> >> > >CPU
> >> > >> > usage
> >> > >> > > > > that
> >> > >> > > > > > > >> renders JMeter unusable.
CPU usage ramps quickly
> >up
> >> > >and
> >> > >> after
> >> > >> > > > having
> >> > >> > > > > > > >> ~100-200 concurrent users
or so (in ~30 seconds),
> >it
> >> > >hits
> >> > >> 100%
> >> > >> > > on
> >> > >> > > > my
> >> > >> > > > > > > >> i7-4600. Shortly after,
whole JMeter freezes and
> >I
> >> > >have to
> >> > >> > kill
> >> > >> > > > it.
> >> > >> > > > > > > >>
> >> > >> > > > > > > >> On 2.13 everything goes
smoothly (with default
> >> > >settings).
> >> > >> CPU
> >> > >> > > > usage
> >> > >> > > > > > > stays
> >> > >> > > > > > > >> at ~30% even with 500 users.
> >> > >> > > > > > > >>
> >> > >> > > > > > > >> The experience is similar
on both Windows 10 and
> >Linux
> >> > >> (Ubuntu
> >> > >> > > > 14.04
> >> > >> > > > > > LTS
> >> > >> > > > > > > >> guest in Virtualbox).
> >> > >> > > > > > > >>
> >> > >> > > > > > > >> The problems I saw in JMeter
3:
> >> > >> > > > > > > >>
> >> > >> > > > > > > >> - java.net.SocketException:
Permission denied:
> >connect
> >> > >> > > > > > > >> - After a while it runs
out of heap space
> >> > >> > > > > > > >>
> >> > >> > > > > > > >> For "Permission denied:
connect" I tried to set
> >> > >> > > > > > > >> java.net.preferIPv4Stack=true
> >> > >> > > > > > > >> (in a few places to make
sure it would really be
> >in
> >> > >use) and
> >> > >> > > tried
> >> > >> > > > > to
> >> > >> > > > > > > run
> >> > >> > > > > > > >> JMeter as administrator
(in Win). No effect.
> >> > >> > > > > > > >>
> >> > >> > > > > > > >> I also tried increasing
heap size, but that
> >didn't
> >> > >help
> >> > >> > either.
> >> > >> > > > And
> >> > >> > > > > I
> >> > >> > > > > > > >> think
> >> > >> > > > > > > >> memory size is not the
root cause here - rather
> >the
> >> > >result.
> >> > >> > > > > > > >>
> >> > >> > > > > > > >> I am normally connected
to VPN but I also tried
> >> > >disabling
> >> > >> it.
> >> > >> > No
> >> > >> > > > > > effect.
> >> > >> > > > > > > >>
> >> > >> > > > > > > >> I tried disabling with
and without IPv6 enabled
> >in
> >> > >Windows
> >> > >> > > network
> >> > >> > > > > > > >> connection settings. No
effect.
> >> > >> > > > > > > >>
> >> > >> > > > > > > >> I am running a few extra
JMeter plugins. Tried
> >cutting
> >> > >my
> >> > >> test
> >> > >> > > > > > scenario
> >> > >> > > > > > > to
> >> > >> > > > > > > >> minimum. No effect.
> >> > >> > > > > > > >>
> >> > >> > > > > > > >> I am running anti-virus
software that I cannot
> >> > >disable, due
> >> > >> to
> >> > >> > > > this
> >> > >> > > > > > > being
> >> > >> > > > > > > >> forced by IT department.
However, it's not a
> >problem
> >> > >with
> >> > >> 2.13
> >> > >> > > so
> >> > >> > > > I
> >> > >> > > > > > > assume
> >> > >> > > > > > > >> it shouldn't be a problem
here either.
> >> > >> > > > > > > >>
> >> > >> > > > > > > >> FINALLY, once I set HTTP
request implementation
> >from
> >> > >Apache
> >> > >> > > > > > HttpClient 4
> >> > >> > > > > > > >> to
> >> > >> > > > > > > >> HttpClient 3.1 or JAVA,
the problem vanishes -
> >CPU
> >> > >stays
> >> > >> where
> >> > >> > > it
> >> > >> > > > > did
> >> > >> > > > > > > with
> >> > >> > > > > > > >> JMeter 2.13.
> >> > >> > > > > > > >>
> >> > >> > > > > > > >> I'm running Oracle JVM
8 (update 121 on Win,
> >update
> >> > >111 on
> >> > >> > > Linux).
> >> > >> > > > > > > >>
> >> > >> > > > > > > >> Why does (the recommended)
HttpClient 4 cause
> >problems
> >> > >here?
> >> > >> > Is
> >> > >> > > > > this a
> >> > >> > > > > > > >> know
> >> > >> > > > > > > >> bug/issue? JMeter 3.x is
running newer version
> >> > >compared to
> >> > >> > 2.13,
> >> > >> > > > but
> >> > >> > > > > > > what
> >> > >> > > > > > > >> change in HttpClient 4
might cause the problems
> >> > >described?
> >> > >> > > > > > > >>
> >> > >> > > > > > > >> Any ideas?
> >> > >> > > > > > > >>
> >> > >> > > > > > > >> Regards,
> >> > >> > > > > > > >> Tuukka
> >> > >> > > > > > > >>
> >> > >> > > > > > > >
> >> > >> > > > > > > >
> >> > >> > > > > > > > --
> >> > >> > > > > > > > Cordialement.
> >> > >> > > > > > > > Philippe Mouawad.
> >> > >> > > > > > > >
> >> > >> > > > > > > >
> >> > >> > > > > > > >
> >> > >> > > > > > > >
> >> > >> > > > > > >
> >> > >> > > > > > > --
> >> > >> > > > > > > Cordialement.
> >> > >> > > > > > > Philippe Mouawad.
> >> > >> > > > > > >
> >> > >> > > > > >
> >> > >> > > > >
> >> > >> > > > >
> >> > >> > > > > --
> >> > >> > > > > Cordialement.
> >> > >> > > > > Philippe Mouawad.
> >> > >> > > > >
> >> > >> > > >
> >> > >> > >
> >> > >> >
> >> > >>
> >> > >>
> >> > >>
> >> > >> --
> >> > >> Cordialement.
> >> > >> Philippe Mouawad.
> >> > >>
> >> >
> >> >
> >---------------------------------------------------------------------
> >> > To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
> >> > For additional commands, e-mail: user-help@jmeter.apache.org
> >> >
> >> >
> >>
> >>
> >> --
> >> Cordialement.
> >> Philippe Mouawad.
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message