brooklyn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aledsage <>
Subject [GitHub] brooklyn-docs pull request #122: info on debugging memory usage
Date Mon, 07 Nov 2016 12:18:44 GMT
Github user aledsage commented on a diff in the pull request:
    --- Diff: guide/ops/troubleshooting/ ---
    @@ -0,0 +1,94 @@
    +layout: website-normal
    +title: "Troubleshooting: Monitoring Memory Usage"
    +toc: /guide/toc.json
    +## Memory Usage
    +Brooklyn tries to keep in memory as much history of its activity as possible,
    +for displaying through the UI, so it is normal for it to consume as much memory
    +as it can.  It uses "soft references" so these objects will be cleared if needed,
    +but **it is not a sign of anything unusual if Brooklyn is using all its available memory**.
    +The number of active tasks, CPU usage, thread counts, and 
    +retention of soft reference objects are a much better indication of load.
    +This information can be found by looking in the log for lines containing
    +`brooklyn gc`, such as:
    +    2016-09-16 16:19:43,337 DEBUG o.a.b.c.m.i.BrooklynGarbageCollector [brooklyn-gc]:
brooklyn gc (before) - using 910 MB / 3.76 GB memory; 98% soft-reference maybe retention (of
362); 35 threads; tasks: 0 active, 2 unfinished; 31 remembered, 1013 total submitted) 
    +The soft-reference figure is indicative, but the lower this is, the more
    +the JVM has decided to get rid of items that were desired to be kept but optional.
    +It only tracks some soft-references (those wrapped in `Maybe`),
    +and of course if there are many many such items the JVM will have to get rid
    +of some, so a lower figure does not necessarily mean a problem.
    +Typically however if there's no `OutOfMemoryError` (OOME) reported,
    +there's no problem.
    +If you are concerned about memory usage, or doing evaluation on test environments, 
    +the following method (in the Groovy console) can be invoked to force the system to
    +reclaim as much memory as possible, including *all* soft references:
    +    org.apache.brooklyn.util.javalang.MemoryUsageTracker.forceClearSoftReferences()
    +If things are happy usage should return to a small level.  This is quite disruptive
    +to the system however so use with care.
    +The above method can also be configured to run automatically when memory usage 
    +is detected to hit a certain level.  That can be useful if external policies are
    +being used to warn on high memory usage, and you want to keep some headroom.
    +Many JVM references discourage interfering with its garbage collector, however,
    +so use with care and study the particular JVM you are using.
    +See the class `BrooklynGarbageCollector` for more information.
    +## Investigation of Memory Leaks
    +Design problems of course can cause memory leaks, and due to the nature of the
    +soft references these can be difficult to notice until they are advanced.
    +If the "soft-reference maybe retention" starts to decrease, that can be
    +an early warning.
    +Common problems such as runaway tasks and cyclic dependent configuration will often
    +show their own log errors, so also look for these if there is a performance or memory
    +You should also note the task counts in the `brooklyn gc` messages described above,
    +and if there are an exceptional number of tasks or tasks are not clearing,
    +other log messages will describe what is happening, and the in-product task
    +view can indicate issues.  `jstack` can also be useful if it is a task problem.
    --- End diff --
    Also worth recommending the use of `jmap -histo:live <pid>` to see how many live
instances of each type there are.

If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at or file a JIRA ticket
with INFRA.

View raw message