brooklyn-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
Subject [2/3] brooklyn-docs git commit: expand memory usage hints as discussed on PR
Date Mon, 07 Nov 2016 13:29:25 GMT
expand memory usage hints as discussed on PR


Branch: refs/heads/master
Commit: 561ac2a8aa8c2e8a373b17046ede059e4c5002d6
Parents: 8f04abc
Author: Alex Heneveld <>
Authored: Mon Nov 7 13:24:28 2016 +0000
Committer: Alex Heneveld <>
Committed: Mon Nov 7 13:28:06 2016 +0000

 guide/ops/         | 14 +++-
 guide/ops/troubleshooting/ | 94 +++++++++++++++++++-------
 2 files changed, 80 insertions(+), 28 deletions(-)
diff --git a/guide/ops/ b/guide/ops/
index 407de29..6ce34e6 100644
--- a/guide/ops/
+++ b/guide/ops/
@@ -61,19 +61,25 @@ export PATH=$PATH:$BROOKLYN_HOME/usage/dist/target/brooklyn-dist/bin/
 ### Memory Usage
 The amount of memory required by the Brooklyn process depends on the usage 
-- for example the number of entities/VMs under management.
+-- for example the number of entities/VMs under management.
 For a standard Brooklyn deployment, the defaults are to start with 256m, and to grow to 1g
of memory.
 These numbers can be overridden by setting the environment variable `JAVA_OPTS` before launching
-the `brooklyn script`:
+the `brooklyn script`, as follows:
-    JAVA_OPTS=-Xms1g -Xmx1g -XX:MaxPermSize=256m
+    JAVA_OPTS="-Xms1g -Xmx4g -XX:MaxPermSize=256m"
+(On Java 8 and later the last entry has no effect and can be dropped.)
 Brooklyn stores a task history in-memory using [soft references](
 This means that, once the task history is large, Brooklyn will continually use the maximum
 memory. It will only expunge tasks from memory when this space is required for other objects
within the
 Brooklyn process.
+See [Memory Usage](troubleshooting/memory-usage.html) for more information on memory usage
+other suggested `JAVA_OPTS`.
 ### Web Console Bind Address
 The web console will by default bind to It's restricted to if the `--noConsoleSecurity`
flag is used.
@@ -189,3 +195,5 @@ or Swift. It has the following options:
 * `blob --container <containerName> --blob <blobName>`: retrieves the given blob
   (i.e. object), including metadata and its contents.
diff --git a/guide/ops/troubleshooting/ b/guide/ops/troubleshooting/
index cfadaef..bb95342 100644
--- a/guide/ops/troubleshooting/
+++ b/guide/ops/troubleshooting/
@@ -26,57 +26,86 @@ of some, so a lower figure does not necessarily mean a problem.
 Typically however if there's no `OutOfMemoryError` (OOME) reported,
 there's no problem.
+## Problem Indicators and Resolutions
+Two things that *do* normally indicate a problem with memory are:
+* `OutOfMemoryError` exceptions being thrown
+* Memory usage high *and* CPU high, where the CPU is spent doing full garbage collection
+One possible cause is the JVM doing a poorly-selected GC strategy,
+as described in [Oracle Java bug 6912889](
+This can be confirmed by running the "analyzing soft reference usage" technique below;
+memory should shrink dramatically then increase until the problem recurs.
+This can be fixed by passing `-XX:SoftRefLRUPolicyMSPerMB=1` to the JVM,
+as described in [Brooklyn issue 375](
+Other common JVM options include `-Xms256m -Xmx1g -XX:MaxPermSize=256m`
+(depending on JVM provider and version) to set the right balance of memory allocation.
+In some cases a larger `-Xmx` value may simply be the fix
+(but this should not be the case unless many or large blueprints are being used).
+If the problem is not with soft references but with real memory usage,
+the culprit is likely a memory leak, typically in blueprint design.
+An early warning of this situation is the "soft-reference maybe retention" level decreasing.
+In these situations, follow the steps as described below for "Investigating Leaks".
+## Analyzing Soft Reference Usage
 If you are concerned about memory usage, or doing evaluation on test environments, 
 the following method (in the Groovy console) can be invoked to force the system to
 reclaim as much memory as possible, including *all* soft references:
-If things are happy usage should return to a small level.  This is quite disruptive
-to the system however so use with care.
+In good situations, memory usage should return to a small level.  
+This call can be disruptive to the system however so use with care.
 The above method can also be configured to run automatically when memory usage 
 is detected to hit a certain level.  That can be useful if external policies are
 being used to warn on high memory usage, and you want to keep some headroom.
-Many JVM references discourage interfering with its garbage collector, however,
+Many JVM authorities discourage interfering with its garbage collector, however,
 so use with care and study the particular JVM you are using.
 See the class `BrooklynGarbageCollector` for more information.
-## Investigation of Memory Leaks
-Design problems of course can cause memory leaks, and due to the nature of the
-soft references these can be difficult to notice until they are advanced.
-If the "soft-reference maybe retention" starts to decrease, that can be
-an early warning.
+## Investigating Leaks
-Common problems such as runaway tasks and cyclic dependent configuration will often
-show their own log errors, so also look for these if there is a performance or memory problem.
+If a memory leak is found, the first place to look should be the WARN/ERROR logs.
+Many common causes of leaks, including as runaway tasks and cyclic dependent configuration,
+will show their own log errors prior to the memory error.
 You should also note the task counts in the `brooklyn gc` messages described above,
 and if there are an exceptional number of tasks or tasks are not clearing,
 other log messages will describe what is happening, and the in-product task
-view can indicate issues.  `jstack` can also be useful if it is a task problem.
+view can indicate issues. 
 Sometimes slow leaks can occur if blueprints do not clean up entities or locations.
 These can be diagnosed by noting the number of files written to the persistence location,
 if persistence is being used.  Deploying then destroying a blueprint should not leave
 anything behind in the persistence directory.
-More subtle problems can occur and these can be more difficult to pin down.
-Where these have been encountered, we have tried to improve logging and early identification,
-so please do ask what other log `grep` patterns can be useful in certain situations.
-And if you find issues, let us know so we can add them to what we monitor.
+Where problems have been encountered in the past, we have resolved them and/or
+worked to improve logging and early identification.
+Please report any issues so that we can improve this further.
+In many cases we can also give advice on what other log `grep` patterns can be useful.
-If there's a problem you really can't solve, a memory profiler such as VisualVM or Eclipse
-is the standard way to investigate.  If a heap dump was generated on the OOME
-(most JVMs can be configured to generate that), 
-the profiler can load it and investigate the state of the system.
-These can also connect to running systems and be used to investigate instances and growth.
-Monitoring these systems while live can be difficult because
-it will often include many soft and weak references that mask the
-source of a leak.  Common such items include:
+### Standard Java Techniques
+Useful standard Java techniques for tracking memory leaks include:
+* `jstack <pid>` to see what tasks are running
+* `jmap -histo:live <pid>` to see what objects are using memory (see below)
+* Memory profilers such as VisualVM or Eclipse MAT, either connected to a running system
+  against a heap dump generated on an OOME
+More information is available on [the Oracle Java web site](
+Note that some of the above techniques will often include soft and weak references that are
+to the problem (and will be cleared on an OOME). Objects that may be cached in that way include:
 * `BasicConfigKey` (used for the web server and many blueprints)
 * `DslComponent` and `*Task` (used for Brooklyn activities and dependent configuration)
@@ -89,6 +118,21 @@ and look just after, because that will clear all non-essential data from
 (The `forceClearSoftReferences()` actually works by triggering an OOME, in as safe 
 a way as possible.)
-If leaked items are found, the profiler will normally let you see their content
+If leaked items are found, a profiler will normally let you see their content
 and walk backwards along their references to find out why they are being retained.
+### Summary of Techniques
+The following sequence of techniques is a common approach to investigating and fixing memory
+* Note the log lines about `brooklyn gc`, including memory and tasks
+* Do not assume high memory usage alone is an error, as soft reference caches are deliberate;

+  use `forceClearSoftReferences()` to clear these
+* Note any WARN/ERROR messages in the log
+* Tune JVM memory allocation and GC
+* Look for leaking locations or references by creating then destroying a blueprint
+* Use standard JVM profilers
+* Inform the Apache Brooklyn community

View raw message