brooklyn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (BROOKLYN-375) Brooklyn intermittently uses high CPU levels and becomes unresponsive
Date Mon, 07 Nov 2016 12:16:58 GMT

    [ https://issues.apache.org/jira/browse/BROOKLYN-375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15644020#comment-15644020
] 

ASF GitHub Bot commented on BROOKLYN-375:
-----------------------------------------

Github user aledsage commented on a diff in the pull request:

    https://github.com/apache/brooklyn-docs/pull/122#discussion_r86764032
  
    --- Diff: guide/ops/troubleshooting/memory-usage.md ---
    @@ -0,0 +1,94 @@
    +---
    +layout: website-normal
    +title: "Troubleshooting: Monitoring Memory Usage"
    +toc: /guide/toc.json
    +---
    +
    +## Memory Usage
    +
    +Brooklyn tries to keep in memory as much history of its activity as possible,
    +for displaying through the UI, so it is normal for it to consume as much memory
    +as it can.  It uses "soft references" so these objects will be cleared if needed,
    +but **it is not a sign of anything unusual if Brooklyn is using all its available memory**.
    +
    +The number of active tasks, CPU usage, thread counts, and 
    +retention of soft reference objects are a much better indication of load.
    +This information can be found by looking in the log for lines containing
    +`brooklyn gc`, such as:
    +
    +    2016-09-16 16:19:43,337 DEBUG o.a.b.c.m.i.BrooklynGarbageCollector [brooklyn-gc]:
brooklyn gc (before) - using 910 MB / 3.76 GB memory; 98% soft-reference maybe retention (of
362); 35 threads; tasks: 0 active, 2 unfinished; 31 remembered, 1013 total submitted) 
    +
    +The soft-reference figure is indicative, but the lower this is, the more
    +the JVM has decided to get rid of items that were desired to be kept but optional.
    +It only tracks some soft-references (those wrapped in `Maybe`),
    +and of course if there are many many such items the JVM will have to get rid
    +of some, so a lower figure does not necessarily mean a problem.
    +Typically however if there's no `OutOfMemoryError` (OOME) reported,
    +there's no problem.
    +
    +If you are concerned about memory usage, or doing evaluation on test environments, 
    +the following method (in the Groovy console) can be invoked to force the system to
    +reclaim as much memory as possible, including *all* soft references:
    +
    +    org.apache.brooklyn.util.javalang.MemoryUsageTracker.forceClearSoftReferences()
    +
    +If things are happy usage should return to a small level.  This is quite disruptive
    +to the system however so use with care.
    +
    +The above method can also be configured to run automatically when memory usage 
    --- End diff --
    
    I'd go further and say that we discourage this for any production use-case (because it
has not had expensive testing).
    
    A better solution for production use-cases is to use`-XX:SoftRefLRUPolicyMSPerMB=1`. See
the discussion in https://issues.apache.org/jira/browse/BROOKLYN-375. 
    
    We should also describe the symptoms of BROOKLYN-375 - that the CPU usage goes very high,
with almost all of that time being taken up by garbage collection.
    
    We need to do more on BROOKLYN-375 - e.g. implement the cache, to stop using soft references!


> Brooklyn intermittently uses high CPU levels and becomes unresponsive
> ---------------------------------------------------------------------
>
>                 Key: BROOKLYN-375
>                 URL: https://issues.apache.org/jira/browse/BROOKLYN-375
>             Project: Brooklyn
>          Issue Type: Bug
>         Environment: OSX Sierra, Java 1.7
>            Reporter: Duncan Godwin
>
> Intermittently whilst launching a clocker swarm within brooklyn, it uses high CPU levels
and becomes unresponsive. This was noted when testing failover by manally stopping some nodes
with `shutdown -h`.
> [jstack 1|https://gist.github.com/drigodwin/c5946d23ed11350f393d9ba9b80a2a2d]
> [jstack 2|https://gist.github.com/drigodwin/5619b02c0c1d53ceb0c99234d8f0dd96]
> [jclouds.debug.log|https://gist.github.com/drigodwin/365d39d216e6a56c634a5020496ef8f1]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message