jmeter-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tuukka Mustonen <tuukka.musto...@gmail.com>
Subject Re: High CPU utilization in JMeter 3.x with HttpClient 4 leads to freeze
Date Wed, 01 Feb 2017 13:31:41 GMT
@UBIK Load Pack support: is there some other email address that I can send
info to? support@ubikloadpack.com sounds a bit too generic :)

Tuukka


On Wed, Feb 1, 2017 at 3:29 PM, Tuukka Mustonen <tuukka.mustonen@gmail.com>
wrote:

> Hi,
>
>
> *> From thread dump I see you have in GUI mode around  500 threads and
> you using a View Resuls Tree and Summary Report ?*
> *> Does view results Tree only catch errors or also successes ?*
>
> Yeah I add 20 users every 3 seconds up until 500. Then I run 2mins (if
> 2.13 at least).
>
> Yeah, I have Summary Report enabled and View Results Tree, but I only show
> Errors in View Results Tree. So nothing gets bloated there until things
> break apart.
>
> *>    -  try today this exact same plan with this number of threads in
> JMeter 2.13 with a View Results Tree configured as with this *
>
> Yes, everything is equal.
>
>
> *>   - was JMeter 2.13 running with same JVM version as this one ?*
>
> Yeah the JVM is the same Oracle Java 8u121.
>
> I did a couple of re-runs as requested by Philippe. I will send you the
> results in private mail, too.
>
> (Sorry for slowish response time. Gmail thread view didn't show your
> message when I was in "Sent" mode. Now back in Inbox and it shows up...)
>
> Tuukka
>
>
> On Wed, Feb 1, 2017 at 2:26 PM, UBIK LOAD PACK Support <
> support@ubikloadpack.com> wrote:
>
>> Hello,
>> From thread dump I see you have in GUI mode around  500 threads and you
>> using a View Resuls Tree and Summary Report ?
>> Does view results Tree only catch errors or also successes ?
>>
>> Did you:
>>
>>    -  try today this exact same plan with this number of threads in JMeter
>>    2.13 with a View Results Tree configured as with this plan ?
>>    - was JMeter 2.13 running with same JVM version as this one ?
>>
>> Thanks
>>
>> On Wed, Feb 1, 2017 at 12:57 PM, Tuukka Mustonen <
>> tuukka.mustonen@gmail.com>
>> wrote:
>>
>> > Ok I re-ran the test.
>> >
>> > Took Heap Dump just before JMeter froze.
>> >
>> > Took Thread Dumps right after JMeter froze.
>> >
>> > https://drive.google.com/drive/folders/0B1uBdVuLED5NNFNqallGam13MTA
>> >
>> > ...
>> >
>> > For the fun of it, I also tried upgrading to the latest HttpClient 4.5.3
>> > version. No effect.
>> >
>> > I also noticed that if I ping (google) during the perf testing, Windows
>> > ping reports "General failure." when JMeter breaks. After killing it,
>> > connectivity restores and there are replies from the server again. It
>> seems
>> > like JMeter/HttpClient kills my whole network connection...
>> >
>> > Tuukka
>> >
>> >
>> > On Wed, Feb 1, 2017 at 11:44 AM, Philippe Mouawad <
>> > philippe.mouawad@gmail.com> wrote:
>> >
>> > > On Wed, Feb 1, 2017 at 9:55 AM, Tuukka Mustonen <
>> > tuukka.mustonen@gmail.com
>> > > >
>> > > wrote:
>> > >
>> > > > Hi, I'm running solely on Win 10 now. In re-attempts I think I saw
>> > > slightly
>> > > > different behavior in my Linux guest. I'll write more about that
>> later,
>> > > but
>> > > > for simplicity I'm testing only on Windows host now:
>> > > >
>> > > > Thread dumps are downloadable at:
>> > > > https://drive.google.com/drive/folders/0B1uBdVuLED5NS1RFbEFuSS1iejA
>> > > >
>> > > > (Took them with 7-8s interval I think, hope it's ok).
>> > > >
>> > > > I took the thread dumps once JMeter started filling in errors,
>> which is
>> > > > somewhat after my CPU usage hit 100% (and music playback started to
>> > > distort
>> > > > etc.).
>> > > >
>> > > > I also created the head dump, note that I took it after running the
>> > test
>> > > > and after restarting JMeter, as JMeter froze during the test and I
>> had
>> > to
>> > > > kill it. DId you mean I should take it *during* the test?
>> > >
>> > >
>> > > YES.
>> > >
>> > >
>> > > > Anyway, before
>> > > > sharing it I want to make sure it can't contain anything
>> confidential?
>> > > > (parts of my test plan, passwords, environment variables,
>> anything...)
>> > > >
>> > > > Tuukka
>> > > >
>> > > >
>> > > >
>> > > > On Wed, Feb 1, 2017 at 10:10 AM, UBIK LOAD PACK Support <
>> > > > support@ubikloadpack.com> wrote:
>> > > >
>> > > > > Hello,
>> > > > > To investigate the problem:
>> > > > > 1/ to create  thread dumps, install a JDK of same version as
the
>> one
>> > > > > running your JMeter.
>> > > > >
>> > > > > Run until you get the problem, then call using same user as the
>> one
>> > > > running
>> > > > > JMeter 3 times at 5s interval:
>> > > > >  <jdk_home>/bin/jstack -l <pid of JMeter> >>
output_file.txt
>> > > > >
>> > > > > It will also help if you provide the structure of your Test Plan.
>> > > > >
>> > > > > 2/ It would also help if you can provide a Heap Dump. With JMeter
>> > GUI,
>> > > > use
>> > > > > menu Help > Create a Heap Dump
>> > > > > This will create a Heap Dump (rather big file) , it would help
a
>> lot
>> > if
>> > > > you
>> > > > > can make it available.
>> > > > >
>> > > > > Regards
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > > On Wed, Feb 1, 2017 at 9:03 AM, Tuukka Mustonen <
>> > > > tuukka.mustonen@gmail.com
>> > > > > >
>> > > > > wrote:
>> > > > >
>> > > > > > @Philippe:
>> > > > > >
>> > > > > > Yeah, I am normally running in non-GUI mode, but to
>> > > > > > study/inspect/understand this problem I am running in GUI
mode.
>> > > JMeter
>> > > > > 2.13
>> > > > > > works just fine in GUI mode for these tests so JMeter 3.x
should
>> > too.
>> > > > > >
>> > > > > > I'll give the nightly build a try.
>> > > > > >
>> > > > > > Tuukka
>> > > > > >
>> > > > > >
>> > > > > > On Wed, Feb 1, 2017 at 9:59 AM, Philippe Mouawad <
>> > > > > > philippe.mouawad@gmail.com
>> > > > > > > wrote:
>> > > > > >
>> > > > > > > On Wednesday, February 1, 2017, Tuukka Mustonen <
>> > > > > > tuukka.mustonen@gmail.com
>> > > > > > > >
>> > > > > > > wrote:
>> > > > > > >
>> > > > > > > > @Philippe:
>> > > > > > > >
>> > > > > > > > 1. Yeah, I compared them with the same plan, same
machine.
>> > Should
>> > > > be
>> > > > > > > > everything equal.
>> > > > > > > > 2. I tried both the defaults in jmeter.bat (-Xms512m
>> -Xmx512m)
>> > > and
>> > > > as
>> > > > > > > > slightly increased (-Xms512m -Xmx1024m). I didn't
tweak them
>> > much
>> > > > > > > because I
>> > > > > > > > don't believe the issue lies there.
>> > > > > > > > 3. No, I do just simple HTTP GETs (against RESTy
API).
>> > > > > > > > 4. I am not familiar with that but I can try.
If you have
>> > > > > instructions
>> > > > > > to
>> > > > > > > > point to, please do.
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > > 5. I am running with GUI.
>> > > > > > >
>> > > > > > >
>> > > > > > > This is a known anti-pattern .
>> > > > > > > switch to non gui mode if you want correct results.
>> > > > > > > http://jmeter.apache.org/usermanual/best-practices.html
>> > > > > > >
>> > > > > > > http://www.ubik-ingenierie.com/blog/jmeter_performance_
>> > > tuning_tips/
>> > > > > > >
>> > > > > > >
>> > > > > > > > About file sending:
>> > > > > > > >
>> > > > > > > > 6. Nope. I'm just doing simple GETs.
>> > > > > > >
>> > > > > > >  Could you give nightly build a try ?
>> > > > > > > http://jmeter.apache.org/nightly.html
>> > > > > > >
>> > > > > > >
>> > > > > > > > Tuukka
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > On Wed, Feb 1, 2017 at 9:43 AM, Philippe Mouawad
<
>> > > > > > > > philippe.mouawad@gmail.com <javascript:;>
>> > > > > > > > > wrote:
>> > > > > > > >
>> > > > > > > > > Also are you sending files in post body ?
>> > > > > > > > >
>> > > > > > > > > Thank you
>> > > > > > > > >
>> > > > > > > > > On Wednesday, February 1, 2017, Philippe
Mouawad <
>> > > > > > > > > philippe.mouawad@gmail.com <javascript:;>>
>> > > > > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > > Hello,
>> > > > > > > > > > Few questions, please try to answer
them all:
>> > > > > > > > > >
>> > > > > > > > > > 1/ Did you compare 2.13 and 3.1 with
same plan/sape
>> > > machine...
>> > > > > > > > everything
>> > > > > > > > > > equal ?
>> > > > > > > > > > 2/ What are your Xmx settings
>> > > > > > > > > > 3/ does your plan download embedded
resources ?
>> > > > > > > > > > 4/ when issue happens can you make 3
thread dumps at 5
>> > > seconds
>> > > > > > > interval
>> > > > > > > > > > and send file ? (use jstack)
>> > > > > > > > > > 5/ are you running gui or non gui ?
>> > > > > > > > > >
>> > > > > > > > > > Thanks
>> > > > > > > > > > Regards
>> > > > > > > > > >
>> > > > > > > > > > On Wednesday, February 1, 2017, Tuukka
Mustonen <
>> > > > > > > > > tuukka.mustonen@gmail.com <javascript:;>
>> > > > > > > > > > <javascript:_e(%7B%7D,'cvml','tuukka.mustonen@gmail.com
>> > > > > > > <javascript:;>');>>
>> > > > > > > > wrote:
>> > > > > > > > > >
>> > > > > > > > > >> Hi,
>> > > > > > > > > >>
>> > > > > > > > > >> After upgrading to JMeter 3.0 or
3.1 I experience high
>> CPU
>> > > > usage
>> > > > > > > that
>> > > > > > > > > >> renders JMeter unusable. CPU usage
ramps quickly up and
>> > > after
>> > > > > > having
>> > > > > > > > > >> ~100-200 concurrent users or so
(in ~30 seconds), it
>> hits
>> > > 100%
>> > > > > on
>> > > > > > my
>> > > > > > > > > >> i7-4600. Shortly after, whole JMeter
freezes and I
>> have to
>> > > > kill
>> > > > > > it.
>> > > > > > > > > >>
>> > > > > > > > > >> On 2.13 everything goes smoothly
(with default
>> settings).
>> > > CPU
>> > > > > > usage
>> > > > > > > > > stays
>> > > > > > > > > >> at ~30% even with 500 users.
>> > > > > > > > > >>
>> > > > > > > > > >> The experience is similar on both
Windows 10 and Linux
>> > > (Ubuntu
>> > > > > > 14.04
>> > > > > > > > LTS
>> > > > > > > > > >> guest in Virtualbox).
>> > > > > > > > > >>
>> > > > > > > > > >> The problems I saw in JMeter 3:
>> > > > > > > > > >>
>> > > > > > > > > >> - java.net.SocketException: Permission
denied: connect
>> > > > > > > > > >> - After a while it runs out of heap
space
>> > > > > > > > > >>
>> > > > > > > > > >> For "Permission denied: connect"
I tried to set
>> > > > > > > > > >> java.net.preferIPv4Stack=true
>> > > > > > > > > >> (in a few places to make sure it
would really be in
>> use)
>> > and
>> > > > > tried
>> > > > > > > to
>> > > > > > > > > run
>> > > > > > > > > >> JMeter as administrator (in Win).
No effect.
>> > > > > > > > > >>
>> > > > > > > > > >> I also tried increasing heap size,
but that didn't help
>> > > > either.
>> > > > > > And
>> > > > > > > I
>> > > > > > > > > >> think
>> > > > > > > > > >> memory size is not the root cause
here - rather the
>> > result.
>> > > > > > > > > >>
>> > > > > > > > > >> I am normally connected to VPN but
I also tried
>> disabling
>> > > it.
>> > > > No
>> > > > > > > > effect.
>> > > > > > > > > >>
>> > > > > > > > > >> I tried disabling with and without
IPv6 enabled in
>> Windows
>> > > > > network
>> > > > > > > > > >> connection settings. No effect.
>> > > > > > > > > >>
>> > > > > > > > > >> I am running a few extra JMeter
plugins. Tried cutting
>> my
>> > > test
>> > > > > > > > scenario
>> > > > > > > > > to
>> > > > > > > > > >> minimum. No effect.
>> > > > > > > > > >>
>> > > > > > > > > >> I am running anti-virus software
that I cannot disable,
>> > due
>> > > to
>> > > > > > this
>> > > > > > > > > being
>> > > > > > > > > >> forced by IT department. However,
it's not a problem
>> with
>> > > 2.13
>> > > > > so
>> > > > > > I
>> > > > > > > > > assume
>> > > > > > > > > >> it shouldn't be a problem here either.
>> > > > > > > > > >>
>> > > > > > > > > >> FINALLY, once I set HTTP request
implementation from
>> > Apache
>> > > > > > > > HttpClient 4
>> > > > > > > > > >> to
>> > > > > > > > > >> HttpClient 3.1 or JAVA, the problem
vanishes - CPU
>> stays
>> > > where
>> > > > > it
>> > > > > > > did
>> > > > > > > > > with
>> > > > > > > > > >> JMeter 2.13.
>> > > > > > > > > >>
>> > > > > > > > > >> I'm running Oracle JVM 8 (update
121 on Win, update
>> 111 on
>> > > > > Linux).
>> > > > > > > > > >>
>> > > > > > > > > >> Why does (the recommended) HttpClient
4 cause problems
>> > here?
>> > > > Is
>> > > > > > > this a
>> > > > > > > > > >> know
>> > > > > > > > > >> bug/issue? JMeter 3.x is running
newer version
>> compared to
>> > > > 2.13,
>> > > > > > but
>> > > > > > > > > what
>> > > > > > > > > >> change in HttpClient 4 might cause
the problems
>> described?
>> > > > > > > > > >>
>> > > > > > > > > >> Any ideas?
>> > > > > > > > > >>
>> > > > > > > > > >> Regards,
>> > > > > > > > > >> Tuukka
>> > > > > > > > > >>
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > --
>> > > > > > > > > > Cordialement.
>> > > > > > > > > > Philippe Mouawad.
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > --
>> > > > > > > > > Cordialement.
>> > > > > > > > > Philippe Mouawad.
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > --
>> > > > > > > Cordialement.
>> > > > > > > Philippe Mouawad.
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> > >
>> > >
>> > > --
>> > > Cordialement.
>> > > Philippe Mouawad.
>> > >
>> >
>>
>>
>>
>> --
>>
>> Regards
>> Ubik Load Pack <http://ubikloadpack.com> Team
>> Follow us on Twitter <http://twitter.com/ubikloadpack>
>>
>>
>> Cordialement
>> L'équipe Ubik Load Pack <http://ubikloadpack.com>
>> Suivez-nous sur Twitter <http://twitter.com/ubikloadpack>
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message