brooklyn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <>
Subject [jira] [Commented] (BROOKLYN-375) Brooklyn intermittently uses high CPU levels and becomes unresponsive
Date Mon, 07 Nov 2016 12:16:58 GMT


ASF GitHub Bot commented on BROOKLYN-375:

Github user aledsage commented on a diff in the pull request:
    --- Diff: guide/ops/troubleshooting/ ---
    @@ -0,0 +1,94 @@
    +layout: website-normal
    +title: "Troubleshooting: Monitoring Memory Usage"
    +toc: /guide/toc.json
    +## Memory Usage
    +Brooklyn tries to keep in memory as much history of its activity as possible,
    +for displaying through the UI, so it is normal for it to consume as much memory
    +as it can.  It uses "soft references" so these objects will be cleared if needed,
    +but **it is not a sign of anything unusual if Brooklyn is using all its available memory**.
    +The number of active tasks, CPU usage, thread counts, and 
    +retention of soft reference objects are a much better indication of load.
    +This information can be found by looking in the log for lines containing
    +`brooklyn gc`, such as:
    +    2016-09-16 16:19:43,337 DEBUG o.a.b.c.m.i.BrooklynGarbageCollector [brooklyn-gc]:
brooklyn gc (before) - using 910 MB / 3.76 GB memory; 98% soft-reference maybe retention (of
362); 35 threads; tasks: 0 active, 2 unfinished; 31 remembered, 1013 total submitted) 
    +The soft-reference figure is indicative, but the lower this is, the more
    +the JVM has decided to get rid of items that were desired to be kept but optional.
    +It only tracks some soft-references (those wrapped in `Maybe`),
    +and of course if there are many many such items the JVM will have to get rid
    +of some, so a lower figure does not necessarily mean a problem.
    +Typically however if there's no `OutOfMemoryError` (OOME) reported,
    +there's no problem.
    +If you are concerned about memory usage, or doing evaluation on test environments, 
    +the following method (in the Groovy console) can be invoked to force the system to
    +reclaim as much memory as possible, including *all* soft references:
    +    org.apache.brooklyn.util.javalang.MemoryUsageTracker.forceClearSoftReferences()
    +If things are happy usage should return to a small level.  This is quite disruptive
    +to the system however so use with care.
    +The above method can also be configured to run automatically when memory usage 
    --- End diff --
    I'd go further and say that we discourage this for any production use-case (because it
has not had expensive testing).
    A better solution for production use-cases is to use`-XX:SoftRefLRUPolicyMSPerMB=1`. See
the discussion in 
    We should also describe the symptoms of BROOKLYN-375 - that the CPU usage goes very high,
with almost all of that time being taken up by garbage collection.
    We need to do more on BROOKLYN-375 - e.g. implement the cache, to stop using soft references!

> Brooklyn intermittently uses high CPU levels and becomes unresponsive
> ---------------------------------------------------------------------
>                 Key: BROOKLYN-375
>                 URL:
>             Project: Brooklyn
>          Issue Type: Bug
>         Environment: OSX Sierra, Java 1.7
>            Reporter: Duncan Godwin
> Intermittently whilst launching a clocker swarm within brooklyn, it uses high CPU levels
and becomes unresponsive. This was noted when testing failover by manally stopping some nodes
with `shutdown -h`.
> [jstack 1|]
> [jstack 2|]
> [jclouds.debug.log|]

This message was sent by Atlassian JIRA

View raw message