jmeter-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sebb <seb...@gmail.com>
Subject Re: JMeter Performance evolution : My 2 cents
Date Sat, 26 May 2012 10:55:50 GMT
On 26 May 2012 02:00, Philippe Mouawad <philippe.mouawad@gmail.com> wrote:
> Hello,
> I finally managed to insert images and Script:
> http://wiki.apache.org/jmeter/JMeterPerformance

Good work.

However, the test plan fails for me:

Could not get GUI for
org.apache.jmeter.reporters.ResultCollector@1f0b7d3
java.lang.ClassNotFoundException:
kg.apc.jmeter.vizualizers.TransactionsPerSecondGui

Either that Listener needs to be replaced with one of the standard
ones, or the page needs to be amended to mention the requirement.

It would also be useful if the test plan had the extension ".jmx"

> Regards
> Philippe
> On Fri, May 25, 2012 at 1:48 PM, Philippe Mouawad <
> philippe.mouawad@gmail.com> wrote:
>
>> Hello,
>> And are you able to insert images ? or Test Plan ?
>>
>> Sebb, I think I tried with Attachment to insert Plan or Image, it proposes
>> a link type with a protocol, but I don't see how to upload the image or
>> Test Plan.
>>
>> There is a button Insert/Edit image but it is disabled.
>>
>> Regards
>> Philippe
>>
>>
>> On Fri, May 25, 2012 at 12:06 PM, Milamber <milamber@apache.org> wrote:
>>
>>> On Fri, May 25, 2012 at 10:35 AM, sebb <sebbaz@gmail.com> wrote:
>>>
>>> > On 25 May 2012 10:31, Milamber <milamber@apache.org> wrote:
>>> > > Hello,
>>> > >
>>> > > Actuallly, the page
>>> http://wiki.apache.org/jmeter/JMeterPerformancereturns
>>> > > a http 500 error (others pages works)
>>> >
>>> > Works for me at present.
>>> >
>>>
>>> Ok for me too.
>>>
>>>
>>>
>>>
>>> >
>>> > > Milamber
>>> > >
>>> > > ===
>>> > > Internal Server Error
>>> > >
>>> > > The server encountered an internal error or misconfiguration and was
>>> > unable
>>> > > to complete your request.
>>> > >
>>> > > Please contact the server administrator at infrastructure@apache.orgto
>>> > > inform them of the time this error occurred, and the actions you
>>> > performed
>>> > > just before this error.
>>> > >
>>> > > More information about this error may be available in the server error
>>> > log.
>>> > > ------------------------------
>>> > > Apache/2.4.1 (Unix) OpenSSL/1.0.0g Server at wiki.apache.org Port 80
>>> > >
>>> > > ===
>>> > >
>>> > >
>>> > >
>>> > > On Fri, May 25, 2012 at 1:27 AM, sebb <sebbaz@gmail.com> wrote:
>>> > >
>>> > >> On 24 May 2012 22:15, Philippe Mouawad <philippe.mouawad@gmail.com>
>>> > wrote:
>>> > >> > Hello,
>>> > >> > How can I insert image ? button is disabled in GUI mode.
>>> > >> > To insert Test Plan, is it an attachment ?
>>> > >>
>>> > >> Try clicking "Attachments" before starting the edit.
>>> > >>
>>> > >> > Thanks
>>> > >> > Regards
>>> > >> > Philippe
>>> > >> >
>>> > >> > On Wed, May 23, 2012 at 12:22 AM, sebb <sebbaz@gmail.com>
wrote:
>>> > >> >
>>> > >> >> On 22 May 2012 20:18, Philippe Mouawad <
>>> philippe.mouawad@gmail.com>
>>> > >> wrote:
>>> > >> >> > Hello Milamber,
>>> > >> >> > My responses below.
>>> > >> >> > Regards
>>> > >> >> > Philippe
>>> > >> >> >
>>> > >> >> > On Mon, May 21, 2012 at 11:51 PM, Milamber <milamber@apache.org
>>> >
>>> > >> wrote:
>>> > >> >> >
>>> > >> >> >>
>>> > >> >> >>
>>> > >> >> >> Le 21/05/2012 22:12, Philippe Mouawad a ecrit
:
>>> > >> >> >> > Hello,
>>> > >> >> >> > I read recently this little comparison :
>>> > >> >> >> > https://github.com/excilys/gatling/wiki/Benchmarks<
>>> > >> >> >> https://github.com/excilys/gatling/wiki/Benchmark>
>>> > >> >> >> >
>>> > >> >> >> > I reviewed the test plan that was used to
make the test.
>>> > >> >> >> > It seems to me the test is little biased:
>>> > >> >> >> >
>>> > >> >> >> >    - View Results Tree is in test plan
(as it uses a lot of
>>> > memory,
>>> > >> >> it's
>>> > >> >> >> a
>>> > >> >> >> >    big issue)
>>> > >> >> >> >    - View Results in Table (same thing)
>>> > >> >> >> >    - 3 Non Standard JMeter listener, so
It's not pure JMeter:
>>> > >> >> >> >       - jp@gc - Transactions per Second
>>> > >> >> >> >       - jp@gc - Response Times Over Time
>>> > >> >> >> >       - jp@gc - Active Threads Over Time
>>> > >> >> >> >    - Default XML output seems to have
been used , it's
>>> against
>>> > best
>>> > >> >> >> >    practices
>>> > >> >> >> >
>>> > >> >> >> >
>>> > >> >> >> > Besides the following conclusions that seem
to me non
>>> scientific
>>> > >> and
>>> > >> >> >> purely
>>> > >> >> >> > subjective:
>>> > >> >> >> >
>>> > >> >> >> >    - The testers, new to both Gatling
and JMeter found that
>>> > JMeter
>>> > >> was
>>> > >> >> >> >    harder to learn and use than Gatling
to create the
>>> > simulations,
>>> > >> >> >> despite the
>>> > >> >> >> >    use of a proxy.
>>> > >> >> >> >
>>> > >> >> >> >
>>> > >> >> >> > I will only look at other ideas expressed:
>>> > >> >> >> >
>>> > >> >> >> >    - JMeter creates one thread per user
simulated. If there
>>> is
>>> > not
>>> > >> >> enough
>>> > >> >> >> >    memory allocated to the JVM, it can
crash trying to create
>>> > these
>>> > >> >> >> threads
>>> > >> >> >> >       - This need to be detailed, cause
either it fails with
>>> OOM
>>> > >> and
>>> > >> >> it's
>>> > >> >> >> >       not during thread creation, either
it fails with
>>> "unable
>>> > to
>>> > >> >> create
>>> > >> >> >> new
>>> > >> >> >> >       native thread"
>>> > >> >> >> >    - For instance, JMeter could not run
1500 users with 512
>>> MB
>>> > >> (what
>>> > >> >> was
>>> > >> >> >> >    used for Gatling even with 2000 users);
OutOfMemoryErrors
>>> are
>>> > >> >> >> recorded in
>>> > >> >> >> >    the table as *OOM*
>>> > >> >> >> >       - => I made a Test with up to
2000 Threads with 512 m
>>> > without
>>> > >> >> any
>>> > >> >> >> >       crash, it depends on Test and on
application
>>> > >> >> >> >    - Another problem occurred with the
2000 users
>>> simulations;
>>> > it
>>> > >> >> seems
>>> > >> >> >> >    that JMeter can not simulate more than
1514 users
>>> > independently
>>> > >> >> from
>>> > >> >> >> the
>>> > >> >> >> >    memory that was allocated to the JVM
>>> > >> >> >> >       - => I made a Test with up to
2000 Threads with 512 m
>>> > without
>>> > >> >> any
>>> > >> >> >> >       crash, so assertion is false, it
depends on Test and on
>>> > >> >> application
>>> > >> >> >> >
>>> > >> >> >> >
>>> > >> >> >> > As it's difficult to install the application
used for test
>>> (last
>>> > >> >> version
>>> > >> >> >> > does not seem to work as expected) , if
they provide a
>>> working
>>> > WAR
>>> > >> >> >> against
>>> > >> >> >> > a local Postgres DB I will be happy to test
with it.
>>> > >> >> >> > But in current state, application seems
to be packaged for
>>> cloud
>>> > >> or H2
>>> > >> >> >> > local DB, I didn't want to spend too much
time setting up
>>> > >> application
>>> > >> >> as
>>> > >> >> >> I
>>> > >> >> >> > don't know its real status.
>>> > >> >> >> >
>>> > >> >> >> > I just tried to run Test Plan against a
blank tomcat to
>>> verify
>>> > what
>>> > >> >> they
>>> > >> >> >> > say about Thread Creation, I didn't find
any issue on this.
>>> > >> >> >> >
>>> > >> >> >> > So I decided to make a very simple scenario
test on Tomcat
>>> > Examples
>>> > >> >> (It
>>> > >> >> >> > goes to Session Example, adds attribute,
go back to index, go
>>> > back
>>> > >> to
>>> > >> >> >> > Session Example, test contains Response
assertion for each
>>> > >> Request).
>>> > >> >> >> >
>>> > >> >> >>
>>> > >> >> >> Please, indicate the Tomcat version used? it's
the same
>>> machine?
>>> > >> config
>>> > >> >> >> of tomcat server? tuning of JVM for Tomcat? OS?
OS tuning? JVM
>>> > >> editor?
>>> > >> >> >>
>>> > >> >> >> Tomcat apache-tomcat-6.0.24
>>> > >> >> > -Xms256m -Xmx1024m
>>> > >> >> >    <Connector port="8080" protocol="HTTP/1.1"
>>> > >> >> >
>>> > >> >> >
>>> > >> >>
>>> > >>
>>> >
>>> compressableMimeType="text/html,text/xml,text/plain,text/javascript,application/json"
>>> > >> >> >                compression="off"
>>> > >> >> >                socketBuffer="8"
>>> > >> >> >                maxThreads="400"
>>> > >> >> >               connectionTimeout="20000"
>>> > >> >> >               redirectPort="8443" />
>>> > >> >> > Set session timeout in web.xml  to 1min
>>> > >> >> > java version "1.6.0_29"
>>> > >> >> > Java(TM) SE Runtime Environment (build 1.6.0_29-b11-402-10M3527)
>>> > >> >> > Java HotSpot(TM) 64-Bit Server VM (build 20.4-b02-402,
mixed
>>> mode)
>>> > >> >> >
>>> > >> >> > Mac OS 10.6.8
>>> > >> >> > JMeter  and Tomcat on same machine
>>> > >> >> > No particular OS Tuning
>>> > >> >> > 8GO RAM
>>> > >> >> > No swap, nothing runnning except these 2
>>> > >> >> > Tomcat CPU around 5%
>>> > >> >> > Memory around 50 mo
>>> > >> >> >
>>> > >> >> > JVM editor/version for your JMeter? OS? OS Tuning
(TCP tuning?)?
>>> > >> >> >>
>>> > >> >> >> Same config
>>> > >> >> > No TCP tuning
>>> > >> >> >
>>> > >> >> >> Can you post the jmx file?
>>> > >> >> >>
>>> > >> >> >> Will it be accepted on this list ?
>>> > >> >>
>>> > >> >> Best to create a Wiki page and attach the test file there.
>>> > >> >>
>>> > >> >> >
>>> > >> >> >> Milamber
>>> > >> >> >>
>>> > >> >> >>
>>> > >> >> >> > It is not at all representative but it is
a way for me to
>>> check
>>> > >> >> potential
>>> > >> >> >> > issues in JMeter and performance changes
accross versions.
>>> > >> >> >> >
>>> > >> >> >> > I ran the test with 1500 VU using JMeter
2.5.1,  2.7 (current
>>> > >> trunk)
>>> > >> >> with
>>> > >> >> >> > -Xmx512m, 10 minutes run and CSV output
against a local
>>> Tomcat
>>> > (I
>>> > >> >> restard
>>> > >> >> >> > tomcat between tests and control its health):
>>> > >> >> >> >
>>> > >> >> >> >    - I noticed that current trunk version
behaves much
>>> better in
>>> > >> >> terms of
>>> > >> >> >> >    memory than 2.5.1 or 2.6:
>>> > >> >> >> >       - In 2.5.1 :
>>> > >> >> >> >       - GC activity is much higher with
around 5 GC CPU peaks
>>> > >> every 2
>>> > >> >> >> >          minutes,  and 20 FULL GC
of 700 to 800 ms each
>>> > >> >> >> >          - Throughput: 97,71%
>>> > >> >> >> >          - Pauses : 13,69s
>>> > >> >> >> >          - Mém : 391M/min
>>> > >> >> >> >          - Full GC tend to be much
more frequent at end of
>>> test
>>> > >> >> >> >          - 2.7:
>>> > >> >> >> >          -  no GC CPU peak, 1 FULL
GC
>>> > >> >> >> >          - Throughput:98.54
>>> > >> >> >> >          - Pauses : 8.9s
>>> > >> >> >> >          - 1108m /min
>>> > >> >> >> >          - Summary:
>>> > >> >> >> >       - 25.1:
>>> > >> >> >> >          - 164676 samples in 605,1s
>>> > >> >> >> >          - 272,2/s
>>> > >> >> >> >          - Avg:    97
>>> > >> >> >> >          - 2.7:
>>> > >> >> >> >          - 165367 in 605.0s
>>> > >> >> >> >          - 273.3/s
>>> > >> >> >> >          - Avg:   228
>>> > >> >> >> >          - I also noticed results have
a much better look:
>>> > >> >> >> >       - in 2.5.1, Transactions/s are
around 300 / sec during
>>> 4
>>> > >> >> minutes,
>>> > >> >> >> >       then drop to 200/s, go up to 400/s
, then down to 260/s
>>> > and
>>> > >> >> >> > finally 200/sec
>>> > >> >> >> >       - in 2.7, Transactions stay around
300/sec
>>> > >> >> >> >
>>> > >> >> >> >
>>> > >> >> >> >
>>> > >> >> >> > Actions:
>>> > >> >> >> >
>>> > >> >> >> >    - I think it would be useful to have
some reference
>>> > application
>>> > >> on
>>> > >> >> >> which
>>> > >> >> >> >    we could test JMeter behaviour
>>> > >> >> >> >    - What would the best place to put
these comparisons ?
>>> wiki ?
>>> > >> which
>>> > >> >> >> >    indicators should we put ?
>>> > >> >> >> >    - Further testing should be done against
a richer
>>> application
>>> > >> that
>>> > >> >> >> could
>>> > >> >> >> >    be deployed locally
>>> > >> >> >> >
>>> > >> >> >> >
>>> > >> >> >> > Regards
>>> > >> >> >> > Philippe
>>> > >> >> >> >
>>> > >> >> >> >
>>> > >> >> >>
>>> > >> >> >>
>>> > >> >> >
>>> > >> >> >
>>> > >> >> > --
>>> > >> >> > Cordialement.
>>> > >> >> > Philippe Mouawad.
>>> > >> >>
>>> > >> >
>>> > >> >
>>> > >> >
>>> > >> > --
>>> > >> > Cordialement.
>>> > >> > Philippe Mouawad.
>>> > >>
>>> >
>>>
>>
>>
>>
>> --
>> Cordialement.
>> Philippe Mouawad.
>>
>>
>>
>>
>
>
> --
> Cordialement.
> Philippe Mouawad.

Mime
View raw message