jmeter-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sebb <seb...@gmail.com>
Subject Re: OnDemandThreadGroup => Delay startup option on standard ThreadGroup
Date Wed, 15 Aug 2012 19:54:25 GMT
On 15 August 2012 19:40, Milamber <milamber@apache.org> wrote:
>
>
> Le 15/08/2012 09:34, sebb a ecrit :
>
>> On 15 August 2012 08:05, Milamber<milamber@apache.org>  wrote:
>>>
>>>
>>> Le 14/08/2012 21:10, sebb a ecrit :
>>>
>>>> On 14 August 2012 11:32, sebb<sebbaz@gmail.com>   wrote:
>>>>>
>>>>> On 14 August 2012 10:51, Milamber<milamber@apache.org>   wrote:
>>>>>>
>>>>>>
>>>>>> Le 13/08/2012 18:58, sebb a ecrit :
>>>>>>
>>>>>>> On 13 August 2012 17:07, sebb<sebbaz@gmail.com>    wrote:
>>>>>>>>
>>>>>>>> I've made the changes to merge the OnDemand code back into
the
>>>>>>>> standard ThreadGroup.
>>>>>>>>
>>>>>>>> This means it's now trivial for users to switch between the
>>>>>>>> different
>>>>>>>> startup types.
>>>>>>>>
>>>>>>>> I hope I've not broken anything ...
>>>>>>>
>>>>>>> Seems to work OK with a simple test plan of a single Java sampler,
10
>>>>>>> loops.
>>>>>>> Setting the ramp-up to the same as the thread count gives me
a max of
>>>>>>> 3 threads concurrently.
>>>>>>>
>>>>>>> This runs out of memory when run without delayed start and 10000
>>>>>>> threads.
>>>>>>> The same test starts fine and runs for a while with delayed start.
>>>>>>
>>>>>>
>>>>>> I've run a simple test of 2 Java sampler (with 1s sleep time), 10
>>>>>> loops,
>>>>>> with 120000 threads and ramp-up 6000s with jmeter-trunk (on Linux
64
>>>>>> bits
>>>>>> server with 48 cores, sun jdk 6 64bits)
>>>>>>
>>>>>> ===
>>>>>> With delayed start, test works fine.
>>>>>> 2400004 of sampler results
>>>>>>
>>>>>> Max heap size during tests: ~2.6 GB
>>>>>> (jmeter java opts : HEAP="-Xms8g -Xmx8g -XX:+UseConcMarkSweepGC
>>>>>> -Xloggc:/tmp/gclog_`date '+20%y%m%d%H%M%S'`.txt")
>>>>>>
>>>>>> Average number of LWP (Light-weight process): ~500  (~threads inside
>>>>>> the
>>>>>> java process via the linux command 'ps')
>>>>>>
>>>>>> ===
>>>>>> Without delayed start: OutOfMemoryError: unable to create new native
>>>>>> thread.
>>>>>> 45176 of sampler results (test not ended, kill by me after the OOME
>>>>>> notification)
>>>>>>
>>>>>> Max heap size during tests: ~2.3 GB
>>>>>> (jmeter java opts : HEAP="-Xms8g -Xmx8g -XX:+UseConcMarkSweepGC
>>>>>> -Xloggc:/tmp/gclog_`date '+20%y%m%d%H%M%S'`.txt")
>>>>>>
>>>>>> Max number of LWP (Light-weight process): 30396  (~threads inside
the
>>>>>> java
>>>>>> process)
>>>>>>
>>>>>> ===
>>>>>>
>>>>>> Conclusion IMHO: A great behavior for JMeter and tests with a huge
>>>>>> number of
>>>>>> threads (users) with small number of individual loops per user, like
>>>>>> webservice's test
>>>>>
>>>>> Thanks!
>>>>>
>>>>> I'm currently working on delaying the creation of JMeterThread
>>>>> instances.
>>>>> These take quite a bit of memory, and they aren't needed until just
>>>>> before the thread starts.
>>>>>
>>>>> This means moving code from StandardJMeterEngine to ThreadGroup.
>>>>> Quite a lot of changes, but it's looking very promising - many more
>>>>> threads can be started, as memory is then only taken for active
>>>>> threads.
>>>>>
>>>>> I should be able to commit the code later today or tomorrow.
>>>>
>>>> Committed. Seems to be working OK, though I've not tested multiple
>>>> thread groups yet.
>>>>
>>>> When using delayed start, the thread  count should now be limited only
>>>> by the number of concurrent threads.
>>>> I was able to start a test with 10 million threads (though I did not
>>>> let it run for very long).
>>>>
>>>> Assuming that the thread-related data is properly released when the
>>>> thread finishes, there's no reason why such a test should not
>>>> complete.
>>>
>>>
>>>
>>> I've run again the simple test of 2 Java sampler (with 1s sleep time), 10
>>> loops, with 120000 threads and ramp-up 6000s with jmeter-trunk (on Linux
>>> 64
>>> bits server with 48 cores, sun jdk 6 64bits)
>>>
>>>
>>> ===
>>> With delayed start, test works fine.
>>> 2400004 of sampler results
>>>
>>> Max heap size during tests: end with 2.3 GB (without any Full GC)
>>>
>>> (jmeter java opts : HEAP="-Xms8g -Xmx8g -XX:+UseConcMarkSweepGC
>>> -Xloggc:/tmp/gclog_`date '+20%y%m%d%H%M%S'`.txt")
>>>
>>> Average number of LWP (Light-weight process): ~500  (~threads inside the
>>> java process via the linux command 'ps')
>>>
>>> ===
>>> Without delayed start: OutOfMemoryError: unable to create new native
>>> thread.
>>
>> That's to be expected.
>>
>>> 105704 of sampler results (test not ended, kill by me after the OOME
>>> notification)
>>>
>>> Max heap size during tests: ~7.8 GB
>>>
>>> (jmeter java opts : HEAP="-Xms8g -Xmx8g -XX:+UseConcMarkSweepGC
>>> -Xloggc:/tmp/gclog_`date '+20%y%m%d%H%M%S'`.txt")
>>>
>>> Max number of LWP (Light-weight process): 31083  (~threads inside the
>>> java
>>> process)
>>>
>>> ===
>>>
>>> A new test:
>>> same scenario but sleep_time = 100 ms, 2 loops, with 9,600,000 threads
>>> and
>>> ramp-up 7200s, delayed start  (80000 new threads by minute)
>>>
>>> Test have OOME after 50 min
>>> ~7,715,164 of sampler results
>>>
>>> Max heap size during tests: end with 99 GB (after GC) (big OOME ;-))
>>> (jmeter java opts : HEAP="-Xms100g -Xmx100g -XX:NewSize=50g
>>> -XX:MaxNewSize=50g -XX:PermSize=1g -XX:MaxPermSize=20g
>>> -XX:+UseConcMarkSweepGC -Xloggc:/tmp/gclog_`date '+20%y%m%d%H%M%S'`.txt")
>>>
>>> Average number of LWP (Light-weight process): up to 769 (before the first
>>> Full OOME)  (~threads inside the java process via the linux command 'ps')
>>
>> This is a higher concurrent number than before. It would be
>> interesting to know if the test would succeed with a similar count to
>> before.
>>
>> i.e. keep the ratio of thread-count / ramp-up the same.
>
>
> If we keep the ratio (120,000 thread/6000 sec ramp-up), for 9,600,000
> threads, we need a ramp-up of 480,000 seconds (~5,5 days)

How about running the shorter test with the same ratio instead?

> I can make a test with this ratio only for 800 minute (~13,3 hours) and
> 960,000 threads. I think I must keep the first JVM config (heap at 8GB).
> I will send results asap.
>
>
>
>>
>> That should show if there are thread resources that are not being
>> cleaned up when the thread finishes.
>
>
> If the OOME occurs, I will try to get a heap dump.
>
>
>
>
>>
>>> Max TG number before the 1st OOME : Thread started: Thread Group
>>> 1-1875217
>>>
>>> ===
>>> Milamber
>>>
>>>
>>>
>>>
>>>
>>>>>> Milamber
>>>>>>
>>>>>>
>>>>>>> It's much quicker to stop the test as well.
>>>>>>>
>>>>>>> I was able to start a test with 50000 threads, but after that
memory
>>>>>>> was a problem.
>>>>>>> Also, the test takes quite a long time to start up, presumably
>>>>>>> because
>>>>>>> of the array of JMeterThread instances.
>>>>>>>
>>>>>>>> There's probably still some tidying up that could be done.
>>>>>>>>
>>>>>>>> For example, StandardJMeterEngine still creates the JMeterThread
>>>>>>>> instances at the start. It might be better to make the thread
group
>>>>>>>> class responsible for handling these, so the additional objects
are
>>>>>>>> not created until needed.
>>>>>>>
>>>>>>> I think that's what broke my tests with more than about 60000
>>>>>>> threads.
>>>>>>>
>>>>>>> As well as deferring creation of these objects,they need to be
>>>>>>> released once the thread has completed.
>>>>>>>
>>>>>>>> BTW, my French is not up to translating the delayed_start
property
>>>>>>>> entry - please can someone else fix that so the tests no
longer
>>>>>>>> fail?
>>>>>>>> Thanks!
>>>>>>
>>>>>>
>

Mime
View raw message