jmeter-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Milamber <milam...@apache.org>
Subject Re: OnDemandThreadGroup => Delay startup option on standard ThreadGroup
Date Fri, 31 Aug 2012 20:46:57 GMT


Le 18/08/2012 15:12, sebb a ecrit :
> On 18 August 2012 13:02, Milamber<milamber@apache.org>  wrote:
>>
>> Le 15/08/2012 19:54, sebb a ecrit :
>>> [snip]
>>>
>>>>>> 105704 of sampler results (test not ended, kill by me after the OOME
>>>>>> notification)
>>>>>>
>>>>>> Max heap size during tests: ~7.8 GB
>>>>>>
>>>>>> (jmeter java opts : HEAP="-Xms8g -Xmx8g -XX:+UseConcMarkSweepGC
>>>>>> -Xloggc:/tmp/gclog_`date '+20%y%m%d%H%M%S'`.txt")
>>>>>>
>>>>>> Max number of LWP (Light-weight process): 31083  (~threads inside
the
>>>>>> java
>>>>>> process)
>>>>>>
>>>>>> ===
>>>>>>
>>>>>> A new test:
>>>>>> same scenario but sleep_time = 100 ms, 2 loops, with 9,600,000 threads
>>>>>> and
>>>>>> ramp-up 7200s, delayed start  (80000 new threads by minute)
>>>>>>
>>>>>> Test have OOME after 50 min
>>>>>> ~7,715,164 of sampler results
>>>>>>
>>>>>> Max heap size during tests: end with 99 GB (after GC) (big OOME ;-))
>>>>>> (jmeter java opts : HEAP="-Xms100g -Xmx100g -XX:NewSize=50g
>>>>>> -XX:MaxNewSize=50g -XX:PermSize=1g -XX:MaxPermSize=20g
>>>>>> -XX:+UseConcMarkSweepGC -Xloggc:/tmp/gclog_`date
>>>>>> '+20%y%m%d%H%M%S'`.txt")
>>>>>>
>>>>>> Average number of LWP (Light-weight process): up to 769 (before the
>>>>>> first
>>>>>> Full OOME)  (~threads inside the java process via the linux command
>>>>>> 'ps')
>>>>> This is a higher concurrent number than before. It would be
>>>>> interesting to know if the test would succeed with a similar count to
>>>>> before.
>>>>>
>>>>> i.e. keep the ratio of thread-count / ramp-up the same.
>>>>
>>>> If we keep the ratio (120,000 thread/6000 sec ramp-up), for 9,600,000
>>>> threads, we need a ramp-up of 480,000 seconds (~5,5 days)
>>> How about running the shorter test with the same ratio instead?
>>
>> Same thing. a OutOfMemroyError.
>> I've run several tests (Xmx = 100G or 8G or 512 M). Always the OOME.
>>
>> Heap Dump show a huge retained memory in o.a.j.threads.TestCompiler in the
>> private static final Set<ObjectPair>  pairing = new HashSet<ObjectPair>();
>>
>> Memory Analyzer :
>> The class*"org.apache.jmeter.threads.TestCompiler"*, loaded
>> by*"org.apache.jmeter.DynamicClassLoader @ 0xe00272a8"*,
>> occupies*516,943,728 (99.53%)*bytes. The memory is accumulated in one
>> instance of*"java.util.HashMap$Entry[]"*loaded by*"<system class loader>"*.
>>
>> *Keywords*
>> org.apache.jmeter.threads.TestCompiler
>> java.util.HashMap$Entry[]
>> org.apache.jmeter.DynamicClassLoader @ 0xe00272a8
>>
>> ====
>> Inside TestCompiler object (in memory analyzer), we found :
>>
>> Class Name                            | Shallow Heap | Retained Heap
>> ---------------------------------------------------------------------
>> pairing java.util.HashSet @ 0xe013da30|           16 |   516,943,720
>> ---------------------------------------------------------------------
>>
>> The pairing hashset size is 221236 in my test (each element count 56 bytes
>> to 112 bytes)
> Thanks!
>
> I was thinking the pairing Set could be made an instance variable so
> it would disappear at the end of a thread.
>
> However, that might perhaps cause problems for test classes that are
> marked NoThreadClone.
>
> The ObjectPair class equals uses ==, so for per-thread child and
> parent entries, it would make no difference if the Set were an
> instance variable.
>
> However, if both the child and parent classes are shared between
> threads, then it would be possible for one thread to match another's
> ObjectPair.
> It would be necessary to use a static Set to catch this case; however
> I don't know if this case can ever occur.
>
> It might be worth trying the test with a temporary version of TestCompiler:
>   - just remove the "static" qualifier, and comment out the initialise body.
>
> If that makes a big difference it would be worth trying to move most
> of the data to an instance variable.
>
> Meanwhile 'll try and do some more investigation to see if we can drop
> the static qualifier.

Good news:


Since your commit r1378780 (bug 53796), I've can run a very large test 
with a lot of threads successfully!

JMeter trunk, without JVM parameters changes (i.e. Xmx=512MB) except a 
log gc option

Number of threads : 960,000
Ramp up : 48,000
Loop : 10
Delay thread creation enable

Scenario: 2 Java requests with 1 sec sleep time

Results : 19,200,004 lines in csv files (size 2,3 GB)

GC never reach 135MB (used) (arround 28 MB after a GC).



Possible to display a the new response time graph, with a Xmx=2G (need 
1,4 GB)
http://www.milamberspace.net/jmeter/resptime1.png
(great stability)


Milamber

>
>> ====
>>
>> To reproduce this OOME :
>>
>> A simple test :
>>
>> Thread Group with Startup delay, 960000 Threads, 48000 ramp-up, 10 loops
>>     |-- 1 Java request (or HTTP request)
>>     |-- 2 Java request (or HTTP request)
>> -- Simple Writer
>>
>>
>> JVM option : default Xms and Xmx (512M), and modify variable SERVER
>> (optionnal):
>> SERVER="-server -XX:+UseConcMarkSweepGC -Xloggc:/tmp/gclog_`date
>> '+20%y%m%d%H%M%S'`.txt"
>>
>> make a heap dump (just before or) when the first Full GC come.
>> jmap -F -dump:format=b,file=Jmeter.hprof<PID>
>> (or wait the heap dump from the option "-XX:+HeapDumpOnOutOfMemoryError")
>>
>> Milamber.
>>
>>
>>
>>
>>


Mime
View raw message