jmeter-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Felix Schumacher <felix.schumac...@internetallee.de>
Subject Re: High CPU utilization in JMeter 3.x with HttpClient 4 leads to freeze
Date Fri, 03 Feb 2017 05:57:16 GMT


Am 2. Februar 2017 06:52:32 MEZ schrieb Tuukka Mustonen <tuukka.mustonen@gmail.com>:
>@Philippe:
>
>0. Alright, I will.
>1. Will do.
>2. Sure.
>3. To let the network connection state restore? Ok, will do.
>
>I am having some other stuff I need to attend today, but I will get
>back to
>this in the afternoon at latest.

Could you also try to set loggerpanel.usejsyntaxtext to false in your settings? 

Are there more log entries, when you use httpclient 4 instead of Java?

Vladimir has found signs for a event congestion in the awt event queue, that show in your
threaddumps.

Regards,
 Felix

>
>Tuukka
>
>
>On Wed, Feb 1, 2017 at 11:54 PM, Philippe Mouawad <
>philippe.mouawad@gmail.com> wrote:
>
>> Hello,
>> I've created this bugzilla for investigation and to share with you
>the plan
>> I have used to run my own tests:
>> - https://bz.apache.org/bugzilla/show_bug.cgi?id=60676
>>
>> I use a Tomcat 8.5.11 with a localhost and a certificate generated
>and auto
>> signed.
>>
>> So here is what I get on a Mac Book Pro Mid 2015 OSX El Capitan 2.5
>Ghz
>> Intel Core i7
>> Java 8 u121
>> -Xmx512m -Xms512m
>> Test in GUI mode (ANTI-PATTERN)
>>
>> 2.13:
>> - CPU is at 100%
>> - No error
>> - No leak
>>
>> 3.2 nightly with default settings :
>> - CPU is at 100%
>> - No error
>> - No leak
>>
>> 3.2 nightly with this in user.properties :
>> httpclient4.time_to_live=0
>> httpclient4.validate_after_inactivity=0
>>
>> - CPU is at 100%
>> - No error
>> - No leak
>>
>> I attached the Heap Graphs to bugzilla and the plan.
>>
>> So for now I don't  reproduce your issue but I am not in the same
>> configuration as I am not hitting the same website.
>>
>> So
>> 0/ Could you provide by running only 5 requests, the response header
>that
>> your website returns ?
>> 1/ Could you try my test plan by replacing with your target host and
>see if
>> behaviour is the same ?
>> 2/ Ensure nothing is running (No Spotify :-) ) except your JMeter and
>> monitor network as per Felix note
>> 3/ Run 1 test after the other but ensure you wait a bit between each
>one
>>
>>
>> Thanks
>>
>>
>> On Wed, Feb 1, 2017 at 10:00 PM, Felix Schumacher <
>> felix.schumacher@internetallee.de> wrote:
>>
>> >
>> >
>> > Am 1. Februar 2017 12:57:01 MEZ schrieb Tuukka Mustonen <
>> > tuukka.mustonen@gmail.com>:
>> > >Ok I re-ran the test.
>> > >
>> > >Took Heap Dump just before JMeter froze.
>> > >
>> > >Took Thread Dumps right after JMeter froze.
>> > >
>> >
>>https://drive.google.com/drive/folders/0B1uBdVuLED5NNFNqallGam13MTA
>> >
>> > That folder seems to be empty now.
>> >
>> > Are you running the tests with a security manager enabled?
>> >
>> > Can you share the test or at least the used requests?
>> >
>> > Are you using a proxy and if so, how is it configured?
>> >
>> > Is jmeter running locally, or over something like rdp?
>> >
>> > Can you look at the used network connections like netstat -etapn
>would do
>> > on Linux?
>> >
>> > Regards,
>> >  Felix
>> >
>> > >
>> > >...
>> > >
>> > >For the fun of it, I also tried upgrading to the latest HttpClient
>> > >4.5.3
>> > >version. No effect.
>> > >
>> > >I also noticed that if I ping (google) during the perf testing,
>Windows
>> > >ping reports "General failure." when JMeter breaks. After killing
>it,
>> > >connectivity restores and there are replies from the server again.
>It
>> > >seems
>> > >like JMeter/HttpClient kills my whole network connection...
>> > >
>> > >Tuukka
>> > >
>> > >
>> > >On Wed, Feb 1, 2017 at 11:44 AM, Philippe Mouawad <
>> > >philippe.mouawad@gmail.com> wrote:
>> > >
>> > >> On Wed, Feb 1, 2017 at 9:55 AM, Tuukka Mustonen
>> > ><tuukka.mustonen@gmail.com
>> > >> >
>> > >> wrote:
>> > >>
>> > >> > Hi, I'm running solely on Win 10 now. In re-attempts I think I
>saw
>> > >> slightly
>> > >> > different behavior in my Linux guest. I'll write more about
>that
>> > >later,
>> > >> but
>> > >> > for simplicity I'm testing only on Windows host now:
>> > >> >
>> > >> > Thread dumps are downloadable at:
>> > >> >
>https://drive.google.com/drive/folders/0B1uBdVuLED5NS1RFbEFuSS1iejA
>> > >> >
>> > >> > (Took them with 7-8s interval I think, hope it's ok).
>> > >> >
>> > >> > I took the thread dumps once JMeter started filling in errors,
>> > >which is
>> > >> > somewhat after my CPU usage hit 100% (and music playback
>started to
>> > >> distort
>> > >> > etc.).
>> > >> >
>> > >> > I also created the head dump, note that I took it after
>running the
>> > >test
>> > >> > and after restarting JMeter, as JMeter froze during the test
>and I
>> > >had to
>> > >> > kill it. DId you mean I should take it *during* the test?
>> > >>
>> > >>
>> > >> YES.
>> > >>
>> > >>
>> > >> > Anyway, before
>> > >> > sharing it I want to make sure it can't contain anything
>> > >confidential?
>> > >> > (parts of my test plan, passwords, environment variables,
>> > >anything...)
>> > >> >
>> > >> > Tuukka
>> > >> >
>> > >> >
>> > >> >
>> > >> > On Wed, Feb 1, 2017 at 10:10 AM, UBIK LOAD PACK Support <
>> > >> > support@ubikloadpack.com> wrote:
>> > >> >
>> > >> > > Hello,
>> > >> > > To investigate the problem:
>> > >> > > 1/ to create  thread dumps, install a JDK of same version
as
>the
>> > >one
>> > >> > > running your JMeter.
>> > >> > >
>> > >> > > Run until you get the problem, then call using same user
as
>the
>> > >one
>> > >> > running
>> > >> > > JMeter 3 times at 5s interval:
>> > >> > >  <jdk_home>/bin/jstack -l <pid of JMeter> >>
output_file.txt
>> > >> > >
>> > >> > > It will also help if you provide the structure of your Test
>Plan.
>> > >> > >
>> > >> > > 2/ It would also help if you can provide a Heap Dump. With
>JMeter
>> > >GUI,
>> > >> > use
>> > >> > > menu Help > Create a Heap Dump
>> > >> > > This will create a Heap Dump (rather big file) , it would
>help a
>> > >lot if
>> > >> > you
>> > >> > > can make it available.
>> > >> > >
>> > >> > > Regards
>> > >> > >
>> > >> > >
>> > >> > >
>> > >> > >
>> > >> > >
>> > >> > >
>> > >> > >
>> > >> > > On Wed, Feb 1, 2017 at 9:03 AM, Tuukka Mustonen <
>> > >> > tuukka.mustonen@gmail.com
>> > >> > > >
>> > >> > > wrote:
>> > >> > >
>> > >> > > > @Philippe:
>> > >> > > >
>> > >> > > > Yeah, I am normally running in non-GUI mode, but to
>> > >> > > > study/inspect/understand this problem I am running in
GUI
>mode.
>> > >> JMeter
>> > >> > > 2.13
>> > >> > > > works just fine in GUI mode for these tests so JMeter
3.x
>> > >should too.
>> > >> > > >
>> > >> > > > I'll give the nightly build a try.
>> > >> > > >
>> > >> > > > Tuukka
>> > >> > > >
>> > >> > > >
>> > >> > > > On Wed, Feb 1, 2017 at 9:59 AM, Philippe Mouawad <
>> > >> > > > philippe.mouawad@gmail.com
>> > >> > > > > wrote:
>> > >> > > >
>> > >> > > > > On Wednesday, February 1, 2017, Tuukka Mustonen
<
>> > >> > > > tuukka.mustonen@gmail.com
>> > >> > > > > >
>> > >> > > > > wrote:
>> > >> > > > >
>> > >> > > > > > @Philippe:
>> > >> > > > > >
>> > >> > > > > > 1. Yeah, I compared them with the same plan,
same
>machine.
>> > >Should
>> > >> > be
>> > >> > > > > > everything equal.
>> > >> > > > > > 2. I tried both the defaults in jmeter.bat
(-Xms512m
>> > >-Xmx512m)
>> > >> and
>> > >> > as
>> > >> > > > > > slightly increased (-Xms512m -Xmx1024m). I
didn't
>tweak
>> > >them much
>> > >> > > > > because I
>> > >> > > > > > don't believe the issue lies there.
>> > >> > > > > > 3. No, I do just simple HTTP GETs (against
RESTy API).
>> > >> > > > > > 4. I am not familiar with that but I can try.
If you
>have
>> > >> > > instructions
>> > >> > > > to
>> > >> > > > > > point to, please do.
>> > >> > > > >
>> > >> > > > >
>> > >> > > > >
>> > >> > > > >
>> > >> > > > > > 5. I am running with GUI.
>> > >> > > > >
>> > >> > > > >
>> > >> > > > > This is a known anti-pattern .
>> > >> > > > > switch to non gui mode if you want correct results.
>> > >> > > > > http://jmeter.apache.org/usermanual/best-practices.html
>> > >> > > > >
>> > >> > > > > http://www.ubik-ingenierie.com/blog/jmeter_performance_
>> > >> tuning_tips/
>> > >> > > > >
>> > >> > > > >
>> > >> > > > > > About file sending:
>> > >> > > > > >
>> > >> > > > > > 6. Nope. I'm just doing simple GETs.
>> > >> > > > >
>> > >> > > > >  Could you give nightly build a try ?
>> > >> > > > > http://jmeter.apache.org/nightly.html
>> > >> > > > >
>> > >> > > > >
>> > >> > > > > > Tuukka
>> > >> > > > > >
>> > >> > > > > >
>> > >> > > > > > On Wed, Feb 1, 2017 at 9:43 AM, Philippe Mouawad
<
>> > >> > > > > > philippe.mouawad@gmail.com <javascript:;>
>> > >> > > > > > > wrote:
>> > >> > > > > >
>> > >> > > > > > > Also are you sending files in post body
?
>> > >> > > > > > >
>> > >> > > > > > > Thank you
>> > >> > > > > > >
>> > >> > > > > > > On Wednesday, February 1, 2017, Philippe
Mouawad <
>> > >> > > > > > > philippe.mouawad@gmail.com <javascript:;>>
>> > >> > > > > > > wrote:
>> > >> > > > > > >
>> > >> > > > > > > > Hello,
>> > >> > > > > > > > Few questions, please try to answer
them all:
>> > >> > > > > > > >
>> > >> > > > > > > > 1/ Did you compare 2.13 and 3.1
with same
>plan/sape
>> > >> machine...
>> > >> > > > > > everything
>> > >> > > > > > > > equal ?
>> > >> > > > > > > > 2/ What are your Xmx settings
>> > >> > > > > > > > 3/ does your plan download embedded
resources ?
>> > >> > > > > > > > 4/ when issue happens can you make
3 thread dumps
>at 5
>> > >> seconds
>> > >> > > > > interval
>> > >> > > > > > > > and send file ? (use jstack)
>> > >> > > > > > > > 5/ are you running gui or non gui
?
>> > >> > > > > > > >
>> > >> > > > > > > > Thanks
>> > >> > > > > > > > Regards
>> > >> > > > > > > >
>> > >> > > > > > > > On Wednesday, February 1, 2017,
Tuukka Mustonen <
>> > >> > > > > > > tuukka.mustonen@gmail.com <javascript:;>
>> > >> > > > > > > >
><javascript:_e(%7B%7D,'cvml','tuukka.mustonen@gmail.com
>> > >> > > > > <javascript:;>');>>
>> > >> > > > > > wrote:
>> > >> > > > > > > >
>> > >> > > > > > > >> Hi,
>> > >> > > > > > > >>
>> > >> > > > > > > >> After upgrading to JMeter 3.0
or 3.1 I experience
>high
>> > >CPU
>> > >> > usage
>> > >> > > > > that
>> > >> > > > > > > >> renders JMeter unusable. CPU
usage ramps quickly
>up
>> > >and
>> > >> after
>> > >> > > > having
>> > >> > > > > > > >> ~100-200 concurrent users or
so (in ~30 seconds),
>it
>> > >hits
>> > >> 100%
>> > >> > > on
>> > >> > > > my
>> > >> > > > > > > >> i7-4600. Shortly after, whole
JMeter freezes and
>I
>> > >have to
>> > >> > kill
>> > >> > > > it.
>> > >> > > > > > > >>
>> > >> > > > > > > >> On 2.13 everything goes smoothly
(with default
>> > >settings).
>> > >> CPU
>> > >> > > > usage
>> > >> > > > > > > stays
>> > >> > > > > > > >> at ~30% even with 500 users.
>> > >> > > > > > > >>
>> > >> > > > > > > >> The experience is similar on
both Windows 10 and
>Linux
>> > >> (Ubuntu
>> > >> > > > 14.04
>> > >> > > > > > LTS
>> > >> > > > > > > >> guest in Virtualbox).
>> > >> > > > > > > >>
>> > >> > > > > > > >> The problems I saw in JMeter
3:
>> > >> > > > > > > >>
>> > >> > > > > > > >> - java.net.SocketException:
Permission denied:
>connect
>> > >> > > > > > > >> - After a while it runs out
of heap space
>> > >> > > > > > > >>
>> > >> > > > > > > >> For "Permission denied: connect"
I tried to set
>> > >> > > > > > > >> java.net.preferIPv4Stack=true
>> > >> > > > > > > >> (in a few places to make sure
it would really be
>in
>> > >use) and
>> > >> > > tried
>> > >> > > > > to
>> > >> > > > > > > run
>> > >> > > > > > > >> JMeter as administrator (in
Win). No effect.
>> > >> > > > > > > >>
>> > >> > > > > > > >> I also tried increasing heap
size, but that
>didn't
>> > >help
>> > >> > either.
>> > >> > > > And
>> > >> > > > > I
>> > >> > > > > > > >> think
>> > >> > > > > > > >> memory size is not the root
cause here - rather
>the
>> > >result.
>> > >> > > > > > > >>
>> > >> > > > > > > >> I am normally connected to VPN
but I also tried
>> > >disabling
>> > >> it.
>> > >> > No
>> > >> > > > > > effect.
>> > >> > > > > > > >>
>> > >> > > > > > > >> I tried disabling with and without
IPv6 enabled
>in
>> > >Windows
>> > >> > > network
>> > >> > > > > > > >> connection settings. No effect.
>> > >> > > > > > > >>
>> > >> > > > > > > >> I am running a few extra JMeter
plugins. Tried
>cutting
>> > >my
>> > >> test
>> > >> > > > > > scenario
>> > >> > > > > > > to
>> > >> > > > > > > >> minimum. No effect.
>> > >> > > > > > > >>
>> > >> > > > > > > >> I am running anti-virus software
that I cannot
>> > >disable, due
>> > >> to
>> > >> > > > this
>> > >> > > > > > > being
>> > >> > > > > > > >> forced by IT department. However,
it's not a
>problem
>> > >with
>> > >> 2.13
>> > >> > > so
>> > >> > > > I
>> > >> > > > > > > assume
>> > >> > > > > > > >> it shouldn't be a problem here
either.
>> > >> > > > > > > >>
>> > >> > > > > > > >> FINALLY, once I set HTTP request
implementation
>from
>> > >Apache
>> > >> > > > > > HttpClient 4
>> > >> > > > > > > >> to
>> > >> > > > > > > >> HttpClient 3.1 or JAVA, the
problem vanishes -
>CPU
>> > >stays
>> > >> where
>> > >> > > it
>> > >> > > > > did
>> > >> > > > > > > with
>> > >> > > > > > > >> JMeter 2.13.
>> > >> > > > > > > >>
>> > >> > > > > > > >> I'm running Oracle JVM 8 (update
121 on Win,
>update
>> > >111 on
>> > >> > > Linux).
>> > >> > > > > > > >>
>> > >> > > > > > > >> Why does (the recommended) HttpClient
4 cause
>problems
>> > >here?
>> > >> > Is
>> > >> > > > > this a
>> > >> > > > > > > >> know
>> > >> > > > > > > >> bug/issue? JMeter 3.x is running
newer version
>> > >compared to
>> > >> > 2.13,
>> > >> > > > but
>> > >> > > > > > > what
>> > >> > > > > > > >> change in HttpClient 4 might
cause the problems
>> > >described?
>> > >> > > > > > > >>
>> > >> > > > > > > >> Any ideas?
>> > >> > > > > > > >>
>> > >> > > > > > > >> Regards,
>> > >> > > > > > > >> Tuukka
>> > >> > > > > > > >>
>> > >> > > > > > > >
>> > >> > > > > > > >
>> > >> > > > > > > > --
>> > >> > > > > > > > Cordialement.
>> > >> > > > > > > > Philippe Mouawad.
>> > >> > > > > > > >
>> > >> > > > > > > >
>> > >> > > > > > > >
>> > >> > > > > > > >
>> > >> > > > > > >
>> > >> > > > > > > --
>> > >> > > > > > > Cordialement.
>> > >> > > > > > > Philippe Mouawad.
>> > >> > > > > > >
>> > >> > > > > >
>> > >> > > > >
>> > >> > > > >
>> > >> > > > > --
>> > >> > > > > Cordialement.
>> > >> > > > > Philippe Mouawad.
>> > >> > > > >
>> > >> > > >
>> > >> > >
>> > >> >
>> > >>
>> > >>
>> > >>
>> > >> --
>> > >> Cordialement.
>> > >> Philippe Mouawad.
>> > >>
>> >
>> >
>---------------------------------------------------------------------
>> > To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
>> > For additional commands, e-mail: user-help@jmeter.apache.org
>> >
>> >
>>
>>
>> --
>> Cordialement.
>> Philippe Mouawad.
>>

Mime
  • Unnamed multipart/alternative (inline, 7-Bit, 0 bytes)
View raw message