jmeter-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Milamber <milam...@apache.org>
Subject Re: JMeter Performance evolution : My 2 cents
Date Sat, 26 May 2012 20:25:10 GMT
Hello,

Thanks for the wiki page.

I've got the test plan, and I run some tests.

Target:
1 VM on KVM host | 2 Gb RAM | 4 vCPU
Debian 6.0.5 (64 bits)
Tomcat 6.0.35 (from Apache site)
Sun Java 1.6u32 (64bits)
Restart Tomcat before each test
Tomcat with default params.

JMeter machine (another machine) :
8 Gb RAM | 2 core 2.53GHz
Debian 6.0.5  (64 bits)
Sun Java 1.6u32 (64bits)
JMeter only change on JAVA options: -Xloggc:/tmp/gc_jmeterXXX.txt

With the GC log, I've make some stats.

With the original test plan: HttpClient 4:
JMeter_251    Nb GC: 202    Full GC: 17    Time GC: 13.9376210    Full:
11.3386420    Total: 25.2762630
JMeter_26    Nb GC: 216    Full GC: 18    Time GC: 14.6083630    Full:
12.0604050    Total: 26.6687680
JMeter_27    Nb GC: 199    Full GC: 15    Time GC: 11.7238800    Full:
9.5478390    Total: 21.2717190

Stats results HC4:
JMeter 2.5.1: Requests 163917 in 604.8s =  271.0/s Avg:   148
JMeter 2.6: Requests 163208 in 605.0s =  269.7/s Avg:   155
JMeter 2.7RC3: 164035 in 605.1s =  271.1/s Avg:   159

Changes HttpClient to 3.1:
JMeter_251    Nb GC: 136    Full GC: 1    Time GC: 8.6780490    Full:
.0254120    Total: 8.7034610
JMeter_26    Nb GC: 146    Full GC: 1    Time GC: 9.0567930    Full:
.0305490    Total: 9.0873420
JMeter_27    Nb GC: 137    Full GC: 1    Time GC: 6.9212530    Full:
.0282870    Total: 6.9495400

Stats results HC3.1:
JMeter 2.5.1: Requests 165103 in 605.1s =  272.9/s Avg:   233
JMeter 2.6: Requests 165063 in 605.1s =  272.8/s Avg:   214
JMeter 2.7RC3: Requests 164860 in 604.5s =  272.7/s Avg:   180

Changes to Java http client:
JMeter_251    Nb GC: 138    Full GC: 1    Time GC: 8.1128620    Full:
.0253190    Total: 8.1381810
JMeter_26    Nb GC: 149    Full GC: 2    Time GC: 8.4811800    Full:
.6600440    Total: 9.1412240
JMeter_27    Nb GC: 137    Full GC: 1    Time GC: 6.4912220    Full:
.0283680    Total: 6.5195900

Stats results HJ:
JMeter 2.5.1: Requests 164966 in 605.0s =  272.7/s Avg:   164
JMeter 2.6: Requests 165069 in 605.2s =  272.8/s Avg:   198
JMeter 2.7RC3: Requests 165233 in 605.0s =  273.1/s Avg:   213


* JMeter 2.7(RC3) and HttpJava requests seems the best with the GC
(HC3.1 too)
* HttpJava and HC 3.1 seems the best with GC and req/s
* HC 4 seems have the best results with the time average
* HC 4 seems bring some (big) differences with the java virtual machine.

An investigation on HC4 can be done? perhaps send an email on
HttpComponents team.

Milamber


Le 26/05/2012 02:00, Philippe Mouawad a ecrit :
> Hello,
> I finally managed to insert images and Script:
> http://wiki.apache.org/jmeter/JMeterPerformance
>
> Regards
> Philippe
> On Fri, May 25, 2012 at 1:48 PM, Philippe Mouawad <
> philippe.mouawad@gmail.com> wrote:
>
>   
>> Hello,
>> And are you able to insert images ? or Test Plan ?
>>
>> Sebb, I think I tried with Attachment to insert Plan or Image, it proposes
>> a link type with a protocol, but I don't see how to upload the image or
>> Test Plan.
>>
>> There is a button Insert/Edit image but it is disabled.
>>
>> Regards
>> Philippe
>>
>>
>> On Fri, May 25, 2012 at 12:06 PM, Milamber <milamber@apache.org> wrote:
>>
>>     
>>> On Fri, May 25, 2012 at 10:35 AM, sebb <sebbaz@gmail.com> wrote:
>>>
>>>       
>>>> On 25 May 2012 10:31, Milamber <milamber@apache.org> wrote:
>>>>         
>>>>> Hello,
>>>>>
>>>>> Actuallly, the page
>>>>>           
>>> http://wiki.apache.org/jmeter/JMeterPerformancereturns
>>>       
>>>>> a http 500 error (others pages works)
>>>>>           
>>>> Works for me at present.
>>>>
>>>>         
>>> Ok for me too.
>>>
>>>
>>>
>>>
>>>       
>>>>         
>>>>> Milamber
>>>>>
>>>>> ===
>>>>> Internal Server Error
>>>>>
>>>>> The server encountered an internal error or misconfiguration and was
>>>>>           
>>>> unable
>>>>         
>>>>> to complete your request.
>>>>>
>>>>> Please contact the server administrator at infrastructure@apache.orgto
>>>>> inform them of the time this error occurred, and the actions you
>>>>>           
>>>> performed
>>>>         
>>>>> just before this error.
>>>>>
>>>>> More information about this error may be available in the server error
>>>>>           
>>>> log.
>>>>         
>>>>> ------------------------------
>>>>> Apache/2.4.1 (Unix) OpenSSL/1.0.0g Server at wiki.apache.org Port 80
>>>>>
>>>>> ===
>>>>>
>>>>>
>>>>>
>>>>> On Fri, May 25, 2012 at 1:27 AM, sebb <sebbaz@gmail.com> wrote:
>>>>>
>>>>>           
>>>>>> On 24 May 2012 22:15, Philippe Mouawad <philippe.mouawad@gmail.com>
>>>>>>             
>>>> wrote:
>>>>         
>>>>>>> Hello,
>>>>>>> How can I insert image ? button is disabled in GUI mode.
>>>>>>> To insert Test Plan, is it an attachment ?
>>>>>>>               
>>>>>> Try clicking "Attachments" before starting the edit.
>>>>>>
>>>>>>             
>>>>>>> Thanks
>>>>>>> Regards
>>>>>>> Philippe
>>>>>>>
>>>>>>> On Wed, May 23, 2012 at 12:22 AM, sebb <sebbaz@gmail.com>
wrote:
>>>>>>>
>>>>>>>               
>>>>>>>> On 22 May 2012 20:18, Philippe Mouawad <
>>>>>>>>                 
>>> philippe.mouawad@gmail.com>
>>>       
>>>>>> wrote:
>>>>>>             
>>>>>>>>> Hello Milamber,
>>>>>>>>> My responses below.
>>>>>>>>> Regards
>>>>>>>>> Philippe
>>>>>>>>>
>>>>>>>>> On Mon, May 21, 2012 at 11:51 PM, Milamber <milamber@apache.org
>>>>>>>>>                   
>>>>         
>>>>>> wrote:
>>>>>>             
>>>>>>>>>                   
>>>>>>>>>>
>>>>>>>>>> Le 21/05/2012 22:12, Philippe Mouawad a ecrit :
>>>>>>>>>>                     
>>>>>>>>>>> Hello,
>>>>>>>>>>> I read recently this little comparison :
>>>>>>>>>>> https://github.com/excilys/gatling/wiki/Benchmarks<
>>>>>>>>>>>                       
>>>>>>>>>> https://github.com/excilys/gatling/wiki/Benchmark>
>>>>>>>>>>                     
>>>>>>>>>>> I reviewed the test plan that was used to make
the test.
>>>>>>>>>>> It seems to me the test is little biased:
>>>>>>>>>>>
>>>>>>>>>>>    - View Results Tree is in test plan (as it
uses a lot of
>>>>>>>>>>>                       
>>>> memory,
>>>>         
>>>>>>>> it's
>>>>>>>>                 
>>>>>>>>>> a
>>>>>>>>>>                     
>>>>>>>>>>>    big issue)
>>>>>>>>>>>    - View Results in Table (same thing)
>>>>>>>>>>>    - 3 Non Standard JMeter listener, so It's
not pure JMeter:
>>>>>>>>>>>       - jp@gc - Transactions per Second
>>>>>>>>>>>       - jp@gc - Response Times Over Time
>>>>>>>>>>>       - jp@gc - Active Threads Over Time
>>>>>>>>>>>    - Default XML output seems to have been used
, it's
>>>>>>>>>>>                       
>>> against
>>>       
>>>> best
>>>>         
>>>>>>>>>>>    practices
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Besides the following conclusions that seem to
me non
>>>>>>>>>>>                       
>>> scientific
>>>       
>>>>>> and
>>>>>>             
>>>>>>>>>> purely
>>>>>>>>>>                     
>>>>>>>>>>> subjective:
>>>>>>>>>>>
>>>>>>>>>>>    - The testers, new to both Gatling and JMeter
found that
>>>>>>>>>>>                       
>>>> JMeter
>>>>         
>>>>>> was
>>>>>>             
>>>>>>>>>>>    harder to learn and use than Gatling to create
the
>>>>>>>>>>>                       
>>>> simulations,
>>>>         
>>>>>>>>>> despite the
>>>>>>>>>>                     
>>>>>>>>>>>    use of a proxy.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I will only look at other ideas expressed:
>>>>>>>>>>>
>>>>>>>>>>>    - JMeter creates one thread per user simulated.
If there
>>>>>>>>>>>                       
>>> is
>>>       
>>>> not
>>>>         
>>>>>>>> enough
>>>>>>>>                 
>>>>>>>>>>>    memory allocated to the JVM, it can crash
trying to create
>>>>>>>>>>>                       
>>>> these
>>>>         
>>>>>>>>>> threads
>>>>>>>>>>                     
>>>>>>>>>>>       - This need to be detailed, cause either
it fails with
>>>>>>>>>>>                       
>>> OOM
>>>       
>>>>>> and
>>>>>>             
>>>>>>>> it's
>>>>>>>>                 
>>>>>>>>>>>       not during thread creation, either it fails
with
>>>>>>>>>>>                       
>>> "unable
>>>       
>>>> to
>>>>         
>>>>>>>> create
>>>>>>>>                 
>>>>>>>>>> new
>>>>>>>>>>                     
>>>>>>>>>>>       native thread"
>>>>>>>>>>>    - For instance, JMeter could not run 1500
users with 512
>>>>>>>>>>>                       
>>> MB
>>>       
>>>>>> (what
>>>>>>             
>>>>>>>> was
>>>>>>>>                 
>>>>>>>>>>>    used for Gatling even with 2000 users); OutOfMemoryErrors
>>>>>>>>>>>                       
>>> are
>>>       
>>>>>>>>>> recorded in
>>>>>>>>>>                     
>>>>>>>>>>>    the table as *OOM*
>>>>>>>>>>>       - => I made a Test with up to 2000 Threads
with 512 m
>>>>>>>>>>>                       
>>>> without
>>>>         
>>>>>>>> any
>>>>>>>>                 
>>>>>>>>>>>       crash, it depends on Test and on application
>>>>>>>>>>>    - Another problem occurred with the 2000 users
>>>>>>>>>>>                       
>>> simulations;
>>>       
>>>> it
>>>>         
>>>>>>>> seems
>>>>>>>>                 
>>>>>>>>>>>    that JMeter can not simulate more than 1514
users
>>>>>>>>>>>                       
>>>> independently
>>>>         
>>>>>>>> from
>>>>>>>>                 
>>>>>>>>>> the
>>>>>>>>>>                     
>>>>>>>>>>>    memory that was allocated to the JVM
>>>>>>>>>>>       - => I made a Test with up to 2000 Threads
with 512 m
>>>>>>>>>>>                       
>>>> without
>>>>         
>>>>>>>> any
>>>>>>>>                 
>>>>>>>>>>>       crash, so assertion is false, it depends
on Test and on
>>>>>>>>>>>                       
>>>>>>>> application
>>>>>>>>                 
>>>>>>>>>>>
>>>>>>>>>>> As it's difficult to install the application
used for test
>>>>>>>>>>>                       
>>> (last
>>>       
>>>>>>>> version
>>>>>>>>                 
>>>>>>>>>>> does not seem to work as expected) , if they
provide a
>>>>>>>>>>>                       
>>> working
>>>       
>>>> WAR
>>>>         
>>>>>>>>>> against
>>>>>>>>>>                     
>>>>>>>>>>> a local Postgres DB I will be happy to test with
it.
>>>>>>>>>>> But in current state, application seems to be
packaged for
>>>>>>>>>>>                       
>>> cloud
>>>       
>>>>>> or H2
>>>>>>             
>>>>>>>>>>> local DB, I didn't want to spend too much time
setting up
>>>>>>>>>>>                       
>>>>>> application
>>>>>>             
>>>>>>>> as
>>>>>>>>                 
>>>>>>>>>> I
>>>>>>>>>>                     
>>>>>>>>>>> don't know its real status.
>>>>>>>>>>>
>>>>>>>>>>> I just tried to run Test Plan against a blank
tomcat to
>>>>>>>>>>>                       
>>> verify
>>>       
>>>> what
>>>>         
>>>>>>>> they
>>>>>>>>                 
>>>>>>>>>>> say about Thread Creation, I didn't find any
issue on this.
>>>>>>>>>>>
>>>>>>>>>>> So I decided to make a very simple scenario test
on Tomcat
>>>>>>>>>>>                       
>>>> Examples
>>>>         
>>>>>>>> (It
>>>>>>>>                 
>>>>>>>>>>> goes to Session Example, adds attribute, go back
to index, go
>>>>>>>>>>>                       
>>>> back
>>>>         
>>>>>> to
>>>>>>             
>>>>>>>>>>> Session Example, test contains Response assertion
for each
>>>>>>>>>>>                       
>>>>>> Request).
>>>>>>             
>>>>>>>>>>>                       
>>>>>>>>>> Please, indicate the Tomcat version used? it's the
same
>>>>>>>>>>                     
>>> machine?
>>>       
>>>>>> config
>>>>>>             
>>>>>>>>>> of tomcat server? tuning of JVM for Tomcat? OS? OS
tuning? JVM
>>>>>>>>>>                     
>>>>>> editor?
>>>>>>             
>>>>>>>>>> Tomcat apache-tomcat-6.0.24
>>>>>>>>>>                     
>>>>>>>>> -Xms256m -Xmx1024m
>>>>>>>>>    <Connector port="8080" protocol="HTTP/1.1"
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                   
>>>>>>>>                 
>>>>>>             
>>>>         
>>> compressableMimeType="text/html,text/xml,text/plain,text/javascript,application/json"
>>>       
>>>>>>>>>                compression="off"
>>>>>>>>>                socketBuffer="8"
>>>>>>>>>                maxThreads="400"
>>>>>>>>>               connectionTimeout="20000"
>>>>>>>>>               redirectPort="8443" />
>>>>>>>>> Set session timeout in web.xml  to 1min
>>>>>>>>> java version "1.6.0_29"
>>>>>>>>> Java(TM) SE Runtime Environment (build 1.6.0_29-b11-402-10M3527)
>>>>>>>>> Java HotSpot(TM) 64-Bit Server VM (build 20.4-b02-402,
mixed
>>>>>>>>>                   
>>> mode)
>>>       
>>>>>>>>> Mac OS 10.6.8
>>>>>>>>> JMeter  and Tomcat on same machine
>>>>>>>>> No particular OS Tuning
>>>>>>>>> 8GO RAM
>>>>>>>>> No swap, nothing runnning except these 2
>>>>>>>>> Tomcat CPU around 5%
>>>>>>>>> Memory around 50 mo
>>>>>>>>>
>>>>>>>>> JVM editor/version for your JMeter? OS? OS Tuning (TCP
tuning?)?
>>>>>>>>>                   
>>>>>>>>>> Same config
>>>>>>>>>>                     
>>>>>>>>> No TCP tuning
>>>>>>>>>
>>>>>>>>>                   
>>>>>>>>>> Can you post the jmx file?
>>>>>>>>>>
>>>>>>>>>> Will it be accepted on this list ?
>>>>>>>>>>                     
>>>>>>>> Best to create a Wiki page and attach the test file there.
>>>>>>>>
>>>>>>>>                 
>>>>>>>>>                   
>>>>>>>>>> Milamber
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                     
>>>>>>>>>>> It is not at all representative but it is a way
for me to
>>>>>>>>>>>                       
>>> check
>>>       
>>>>>>>> potential
>>>>>>>>                 
>>>>>>>>>>> issues in JMeter and performance changes accross
versions.
>>>>>>>>>>>
>>>>>>>>>>> I ran the test with 1500 VU using JMeter 2.5.1,
 2.7 (current
>>>>>>>>>>>                       
>>>>>> trunk)
>>>>>>             
>>>>>>>> with
>>>>>>>>                 
>>>>>>>>>>> -Xmx512m, 10 minutes run and CSV output against
a local
>>>>>>>>>>>                       
>>> Tomcat
>>>       
>>>> (I
>>>>         
>>>>>>>> restard
>>>>>>>>                 
>>>>>>>>>>> tomcat between tests and control its health):
>>>>>>>>>>>
>>>>>>>>>>>    - I noticed that current trunk version behaves
much
>>>>>>>>>>>                       
>>> better in
>>>       
>>>>>>>> terms of
>>>>>>>>                 
>>>>>>>>>>>    memory than 2.5.1 or 2.6:
>>>>>>>>>>>       - In 2.5.1 :
>>>>>>>>>>>       - GC activity is much higher with around
5 GC CPU peaks
>>>>>>>>>>>                       
>>>>>> every 2
>>>>>>             
>>>>>>>>>>>          minutes,  and 20 FULL GC of 700 to 800
ms each
>>>>>>>>>>>          - Throughput: 97,71%
>>>>>>>>>>>          - Pauses : 13,69s
>>>>>>>>>>>          - Mém : 391M/min
>>>>>>>>>>>          - Full GC tend to be much more frequent
at end of
>>>>>>>>>>>                       
>>> test
>>>       
>>>>>>>>>>>          - 2.7:
>>>>>>>>>>>          -  no GC CPU peak, 1 FULL GC
>>>>>>>>>>>          - Throughput:98.54
>>>>>>>>>>>          - Pauses : 8.9s
>>>>>>>>>>>          - 1108m /min
>>>>>>>>>>>          - Summary:
>>>>>>>>>>>       - 25.1:
>>>>>>>>>>>          - 164676 samples in 605,1s
>>>>>>>>>>>          - 272,2/s
>>>>>>>>>>>          - Avg:    97
>>>>>>>>>>>          - 2.7:
>>>>>>>>>>>          - 165367 in 605.0s
>>>>>>>>>>>          - 273.3/s
>>>>>>>>>>>          - Avg:   228
>>>>>>>>>>>          - I also noticed results have a much
better look:
>>>>>>>>>>>       - in 2.5.1, Transactions/s are around 300
/ sec during
>>>>>>>>>>>                       
>>> 4
>>>       
>>>>>>>> minutes,
>>>>>>>>                 
>>>>>>>>>>>       then drop to 200/s, go up to 400/s , then
down to 260/s
>>>>>>>>>>>                       
>>>> and
>>>>         
>>>>>>>>>>> finally 200/sec
>>>>>>>>>>>       - in 2.7, Transactions stay around 300/sec
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Actions:
>>>>>>>>>>>
>>>>>>>>>>>    - I think it would be useful to have some
reference
>>>>>>>>>>>                       
>>>> application
>>>>         
>>>>>> on
>>>>>>             
>>>>>>>>>> which
>>>>>>>>>>                     
>>>>>>>>>>>    we could test JMeter behaviour
>>>>>>>>>>>    - What would the best place to put these comparisons
?
>>>>>>>>>>>                       
>>> wiki ?
>>>       
>>>>>> which
>>>>>>             
>>>>>>>>>>>    indicators should we put ?
>>>>>>>>>>>    - Further testing should be done against a
richer
>>>>>>>>>>>                       
>>> application
>>>       
>>>>>> that
>>>>>>             
>>>>>>>>>> could
>>>>>>>>>>                     
>>>>>>>>>>>    be deployed locally
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Regards
>>>>>>>>>>> Philippe
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                       
>>>>>>>>>>
>>>>>>>>>>                     
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Cordialement.
>>>>>>>>> Philippe Mouawad.
>>>>>>>>>                   
>>>>>>>>                 
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Cordialement.
>>>>>>> Philippe Mouawad.
>>>>>>>               
>>>>>>             
>>>>         
>>>       
>>
>>
>> --
>> Cordialement.
>> Philippe Mouawad.
>>
>>
>>
>>
>>     
>
>   


Mime
View raw message