Mailing-List: contact dev-help@jmeter.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@jmeter.apache.org
Message-ID: <502B4A4D.6050409@apache.org>
Date: Wed, 15 Aug 2012 07:05:49 +0000
From: Milamber <milamber@apache.org>
Organization: Apache Software Fondation
User-Agent: Mozilla/5.0 (X11; Linux x86_64;
 rv:10.0.5) Gecko/20120608 Firefox/10.0.5 Iceape/2.7.5
MIME-Version: 1.0
To: dev@jmeter.apache.org
Subject: Re: OnDemandThreadGroup => Delay startup option on standard
 ThreadGroup
References: 
 <CAOGo0VZFKGKGu_VS2RqinWW0ZC+XSYDc4y3_Qj_nKqE8wq8iuw@mail.gmail.com>
 <CAOGo0VY1P-jO0+zUMSaWZZCDt9B_kvVFkh_4Cbb8H-tHvMi=Gw@mail.gmail.com>
 <502A1FB2.30501@apache.org>
 <CAOGo0VZKxFyh2gjchwNpvZO5hH4TgbqZ83q2CqC-jsz8SRqTzg@mail.gmail.com>
 <CAOGo0VZd1bvjvSDXFR-+vUC_3cV5xOmALaww=gs1Jjm+O9Xxdw@mail.gmail.com>
In-Reply-To: 
 <CAOGo0VZd1bvjvSDXFR-+vUC_3cV5xOmALaww=gs1Jjm+O9Xxdw@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit


Le 14/08/2012 21:10, sebb a ecrit :
> On 14 August 2012 11:32, sebb<sebbaz@gmail.com>  wrote:
>> On 14 August 2012 10:51, Milamber<milamber@apache.org>  wrote:
>>>
>>> Le 13/08/2012 18:58, sebb a ecrit :
>>>
>>>> On 13 August 2012 17:07, sebb<sebbaz@gmail.com>   wrote:
>>>>> I've made the changes to merge the OnDemand code back into the
>>>>> standard ThreadGroup.
>>>>>
>>>>> This means it's now trivial for users to switch between the different
>>>>> startup types.
>>>>>
>>>>> I hope I've not broken anything ...
>>>> Seems to work OK with a simple test plan of a single Java sampler, 10
>>>> loops.
>>>> Setting the ramp-up to the same as the thread count gives me a max of
>>>> 3 threads concurrently.
>>>>
>>>> This runs out of memory when run without delayed start and 10000 threads.
>>>> The same test starts fine and runs for a while with delayed start.
>>>
>>> I've run a simple test of 2 Java sampler (with 1s sleep time), 10 loops,
>>> with 120000 threads and ramp-up 6000s with jmeter-trunk (on Linux 64 bits
>>> server with 48 cores, sun jdk 6 64bits)
>>>
>>> ===
>>> With delayed start, test works fine.
>>> 2400004 of sampler results
>>>
>>> Max heap size during tests: ~2.6 GB
>>> (jmeter java opts : HEAP="-Xms8g -Xmx8g -XX:+UseConcMarkSweepGC
>>> -Xloggc:/tmp/gclog_`date '+20%y%m%d%H%M%S'`.txt")
>>>
>>> Average number of LWP (Light-weight process): ~500  (~threads inside the
>>> java process via the linux command 'ps')
>>>
>>> ===
>>> Without delayed start: OutOfMemoryError: unable to create new native thread.
>>> 45176 of sampler results (test not ended, kill by me after the OOME
>>> notification)
>>>
>>> Max heap size during tests: ~2.3 GB
>>> (jmeter java opts : HEAP="-Xms8g -Xmx8g -XX:+UseConcMarkSweepGC
>>> -Xloggc:/tmp/gclog_`date '+20%y%m%d%H%M%S'`.txt")
>>>
>>> Max number of LWP (Light-weight process): 30396  (~threads inside the java
>>> process)
>>>
>>> ===
>>>
>>> Conclusion IMHO: A great behavior for JMeter and tests with a huge number of
>>> threads (users) with small number of individual loops per user, like
>>> webservice's test
>> Thanks!
>>
>> I'm currently working on delaying the creation of JMeterThread instances.
>> These take quite a bit of memory, and they aren't needed until just
>> before the thread starts.
>>
>> This means moving code from StandardJMeterEngine to ThreadGroup.
>> Quite a lot of changes, but it's looking very promising - many more
>> threads can be started, as memory is then only taken for active
>> threads.
>>
>> I should be able to commit the code later today or tomorrow.
> Committed. Seems to be working OK, though I've not tested multiple
> thread groups yet.
>
> When using delayed start, the thread  count should now be limited only
> by the number of concurrent threads.
> I was able to start a test with 10 million threads (though I did not
> let it run for very long).
>
> Assuming that the thread-related data is properly released when the
> thread finishes, there's no reason why such a test should not
> complete.


I've run again the simple test of 2 Java sampler (with 1s sleep time), 
10 loops, with 120000 threads and ramp-up 6000s with jmeter-trunk (on 
Linux 64 bits server with 48 cores, sun jdk 6 64bits)

===
With delayed start, test works fine.
2400004 of sampler results

Max heap size during tests: end with 2.3 GB (without any Full GC)
(jmeter java opts : HEAP="-Xms8g -Xmx8g -XX:+UseConcMarkSweepGC 
-Xloggc:/tmp/gclog_`date '+20%y%m%d%H%M%S'`.txt")

Average number of LWP (Light-weight process): ~500  (~threads inside the 
java process via the linux command 'ps')

===
Without delayed start: OutOfMemoryError: unable to create new native 
thread.
105704 of sampler results (test not ended, kill by me after the OOME 
notification)

Max heap size during tests: ~7.8 GB
(jmeter java opts : HEAP="-Xms8g -Xmx8g -XX:+UseConcMarkSweepGC 
-Xloggc:/tmp/gclog_`date '+20%y%m%d%H%M%S'`.txt")

Max number of LWP (Light-weight process): 31083  (~threads inside the 
java process)

===

A new test:
same scenario but sleep_time = 100 ms, 2 loops, with 9,600,000 threads 
and ramp-up 7200s, delayed start  (80000 new threads by minute)

Test have OOME after 50 min
~7,715,164 of sampler results

Max heap size during tests: end with 99 GB (after GC) (big OOME ;-))
(jmeter java opts : HEAP="-Xms100g -Xmx100g -XX:NewSize=50g 
-XX:MaxNewSize=50g -XX:PermSize=1g -XX:MaxPermSize=20g 
-XX:+UseConcMarkSweepGC -Xloggc:/tmp/gclog_`date '+20%y%m%d%H%M%S'`.txt")

Average number of LWP (Light-weight process): up to 769 (before the 
first Full OOME)  (~threads inside the java process via the linux 
command 'ps')

Max TG number before the 1st OOME : Thread started: Thread Group 1-1875217

===
Milamber


>>> Milamber
>>>
>>>
>>>> It's much quicker to stop the test as well.
>>>>
>>>> I was able to start a test with 50000 threads, but after that memory
>>>> was a problem.
>>>> Also, the test takes quite a long time to start up, presumably because
>>>> of the array of JMeterThread instances.
>>>>
>>>>> There's probably still some tidying up that could be done.
>>>>>
>>>>> For example, StandardJMeterEngine still creates the JMeterThread
>>>>> instances at the start. It might be better to make the thread group
>>>>> class responsible for handling these, so the additional objects are
>>>>> not created until needed.
>>>> I think that's what broke my tests with more than about 60000 threads.
>>>>
>>>> As well as deferring creation of these objects,they need to be
>>>> released once the thread has completed.
>>>>
>>>>> BTW, my French is not up to translating the delayed_start property
>>>>> entry - please can someone else fix that so the tests no longer fail?
>>>>> Thanks!
>>>