cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Evans (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-13006) Disable automatic heap dumps on OOM error
Date Tue, 08 Aug 2017 14:57:00 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-13006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16118421#comment-16118421
] 

Eric Evans commented on CASSANDRA-13006:
----------------------------------------

I think this behavior (invoking {{jmap}} on OOM) is a pretty serious violation to the Element
of Least-surprise.  We already provide mechanisms for passing arguments to the JVM, and TTBMK,
all of them provide some means for dropping a heap dump on out-of-memory.

It definitely caught me be surprise.  We carried over {{-XX:+HeapDumpOnOutOfMemoryError}}
from our 2.2.x environment, only to have Cassandra and the JVM racing to create a dump of
the same name.

Additionally, something about all of this is buggy, because on more than one occasion we've
had Cassandra fork-bombing {{jmap}} processes

{noformat}
● cassandra-b.service - distributed storage system for structured data
   Loaded: loaded (/lib/systemd/system/cassandra-b.service; static)
   Active: active (running) since Sat 2017-08-05 22:32:07 UTC; 23h ago
 Main PID: 25025 (java)
   CGroup: /system.slice/cassandra-b.service
           ├─ 9213 jmap -histo 25025
           ├─ 9214 jmap -histo 25025
           ├─ 9284 jmap -dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof
25025
           ├─ 9285 jmap -dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof
25025
           ├─ 9388 jmap -dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof
25025
           ├─ 9453 jmap -dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof
25025
           ├─ 9519 jmap -dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof
25025
           ├─ 9520 jmap -dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof
25025
           ├─ 9733 jmap -dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof
25025
           ├─ 9735 jmap -dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof
25025
           ├─ 9736 jmap -dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof
25025
           ├─14835 jmap -dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof
25025
           ├─14836 jmap -dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof
25025
           ├─14837 jmap -dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof
25025
           ├─14839 jmap -dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof
25025
           ├─14841 jmap -dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof
25025
           ├─14844 jmap -dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof
25025
           ├─18932 jmap -dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof
25025
           ├─18933 jmap -dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof
25025
           ├─18934 jmap -dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof
25025
           ├─18935 jmap -dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof
25025
           ├─18936 jmap -dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof
25025
           ├─18937 jmap -dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof
25025
           ├─18938 jmap -dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof
25025
           ├─18939 jmap -dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof
25025
           ├─18940 jmap -dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof
25025
           ├─18942 jmap -dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof
25025
           ├─18943 jmap -dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof
25025
           ├─18944 jmap -dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof
25025
           ├─18945 jmap -dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof
25025
           [ ... ]
{noformat}

IMO, the sanest strategy here would be to leave the creation of heap dumps to the JVM.



> Disable automatic heap dumps on OOM error
> -----------------------------------------
>
>                 Key: CASSANDRA-13006
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-13006
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Configuration
>            Reporter: anmols
>            Assignee: Benjamin Lerer
>            Priority: Minor
>             Fix For: 3.0.9
>
>         Attachments: 13006-3.0.9.txt
>
>
> With CASSANDRA-9861, a change was added to enable collecting heap dumps by default if
the process encountered an OOM error. These heap dumps are stored in the Apache Cassandra
home directory unless configured otherwise (see [Cassandra Support Document|https://support.datastax.com/hc/en-us/articles/204225959-Generating-and-Analyzing-Heap-Dumps]
for this feature).
>  
> The creation and storage of heap dumps aides debugging and investigative workflows, but
is not be desirable for a production environment where these heap dumps may occupy a large
amount of disk space and require manual intervention for cleanups. 
>  
> Managing heap dumps on out of memory errors and configuring the paths for these heap
dumps are available as JVM options in JVM. The current behavior conflicts with the Boolean
JVM flag HeapDumpOnOutOfMemoryError. 
>  
> A patch can be proposed here that would make the heap dump on OOM error honor the HeapDumpOnOutOfMemoryError flag.
Users who would want to still generate heap dumps on OOM errors can set the -XX:+HeapDumpOnOutOfMemoryError JVM
option.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org


Mime
View raw message