jmeter-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Philippe Mouawad <philippe.moua...@gmail.com>
Subject OnDemandThreadGroup => Delay startup option on standard ThreadGroup
Date Sat, 18 Aug 2012 19:32:20 GMT
Hello Sebb, Milamber,

Just to be sure I understand did memory leak exist before the optimisation
on JMeterThread memory  (ie creation when needed) (also to have a stable
revision id)
Reading the full discussion, i understand tests where running fine after
sebb reintegrated in ThreadGroup the OnDemandThreaGroup under the delayed
start option  , did I understand well ?
Milamber could you open a bug on nightly bug with your attached test plan ?
I am not sure to understand its content ? Thanks

Sebb, in a later mail you talk about Object pair.
Last time i looked at that part I was wondering what it was used for ? Do
you know ?

Thanks
Regards
Philippe
On Saturday, August 18, 2012, sebb wrote:

> On 18 August 2012 13:02, Milamber <milamber@apache.org> wrote:
> >
> >
> > Le 15/08/2012 19:54, sebb a ecrit :
> >>
> >> [snip]
> >>
> >>>>> 105704 of sampler results (test not ended, kill by me after the
OOME
> >>>>> notification)
> >>>>>
> >>>>> Max heap size during tests: ~7.8 GB
> >>>>>
> >>>>> (jmeter java opts : HEAP="-Xms8g -Xmx8g -XX:+UseConcMarkSweepGC
> >>>>> -Xloggc:/tmp/gclog_`date '+20%y%m%d%H%M%S'`.txt")
> >>>>>
> >>>>> Max number of LWP (Light-weight process): 31083  (~threads inside
the
> >>>>> java
> >>>>> process)
> >>>>>
> >>>>> ===
> >>>>>
> >>>>> A new test:
> >>>>> same scenario but sleep_time = 100 ms, 2 loops, with 9,600,000
> threads
> >>>>> and
> >>>>> ramp-up 7200s, delayed start  (80000 new threads by minute)
> >>>>>
> >>>>> Test have OOME after 50 min
> >>>>> ~7,715,164 of sampler results
> >>>>>
> >>>>> Max heap size during tests: end with 99 GB (after GC) (big OOME
;-))
> >>>>> (jmeter java opts : HEAP="-Xms100g -Xmx100g -XX:NewSize=50g
> >>>>> -XX:MaxNewSize=50g -XX:PermSize=1g -XX:MaxPermSize=20g
> >>>>> -XX:+UseConcMarkSweepGC -Xloggc:/tmp/gclog_`date
> >>>>> '+20%y%m%d%H%M%S'`.txt")
> >>>>>
> >>>>> Average number of LWP (Light-weight process): up to 769 (before
the
> >>>>> first
> >>>>> Full OOME)  (~threads inside the java process via the linux command
> >>>>> 'ps')
> >>>>
> >>>> This is a higher concurrent number than before. It would be
> >>>> interesting to know if the test would succeed with a similar count to
> >>>> before.
> >>>>
> >>>> i.e. keep the ratio of thread-count / ramp-up the same.
> >>>
> >>>
> >>> If we keep the ratio (120,000 thread/6000 sec ramp-up), for 9,600,000
> >>> threads, we need a ramp-up of 480,000 seconds (~5,5 days)
> >>
> >> How about running the shorter test with the same ratio instead?
> >
> >
> > Same thing. a OutOfMemroyError.
> > I've run several tests (Xmx = 100G or 8G or 512 M). Always the OOME.
> >
> > Heap Dump show a huge retained memory in o.a.j.threads.TestCompiler in
> the
> > private static final Set<ObjectPair> pairing = new HashSet<ObjectPair>();
> >
> > Memory Analyzer :
> > The class*"org.apache.jmeter.threads.TestCompiler"*, loaded
> > by*"org.apache.jmeter.DynamicClassLoader @ 0xe00272a8"*,
> > occupies*516,943,728 (99.53%)*bytes. The memory is accumulated in one
> > instance of*"java.util.HashMap$Entry[]"*loaded by*"<system class
> loader>"*.
> >
> > *Keywords*
> > org.apache.jmeter.threads.TestCompiler
> > java.util.HashMap$Entry[]
> > org.apache.jmeter.DynamicClassLoader @ 0xe00272a8
> >
> > ====
> > Inside TestCompiler object (in memory analyzer), we found :
> >
> > Class Name                            | Shallow Heap | Retained Heap
> > ---------------------------------------------------------------------
> > pairing java.util.HashSet @ 0xe013da30|           16 |   516,943,720
> > ---------------------------------------------------------------------
> >
> > The pairing hashset size is 221236 in my test (each element count 56
> bytes
> > to 112 bytes)
>
> Thanks!
>
> I was thinking the pairing Set could be made an instance variable so
> it would disappear at the end of a thread.
>
> However, that might perhaps cause problems for test classes that are
> marked NoThreadClone.
>
> The ObjectPair class equals uses ==, so for per-thread child and
> parent entries, it would make no difference if the Set were an
> instance variable.
>
> However, if both the child and parent classes are shared between
> threads, then it would be possible for one thread to match another's
> ObjectPair.
> It would be necessary to use a static Set to catch this case; however
> I don't know if this case can ever occur.
>
> It might be worth trying the test with a temporary version of TestCompiler:
>  - just remove the "static" qualifier, and comment out the initialise body.
>
> If that makes a big difference it would be worth trying to move most
> of the data to an instance variable.
>
> Meanwhile 'll try and do some more investigation to see if we can drop
> the static qualifier.
>
> > ====
> >
> > To reproduce this OOME :
> >
> > A simple test :
> >
> > Thread Group with Startup delay, 960000 Threads, 48000 ramp-up, 10 loops
> >    |-- 1 Java request (or HTTP request)
> >    |-- 2 Java request (or HTTP request)
> > -- Simple Writer
> >
> >
> > JVM option : default Xms and Xmx (512M), and modify variable SERVER
> > (optionnal):
> > SERVER="-server -XX:+UseConcMarkSweepGC -Xloggc:/tmp/gclog_`date
> > '+20%y%m%d%H%M%S'`.txt"
> >
> > make a heap dump (just before or) when the first Full GC come.
> > jmap -F -dump:format=b,file=Jmeter.hprof <PID>
> > (or wait the heap dump from the option "-XX:+HeapDumpOnOutOfMemoryError")
> >
> > Milamber.
> >
> >
> >
> >
> >
>


-- 
Cordialement.
Philippe Mouawad.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message