Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cassandra.apache.org
Date: Fri, 6 Oct 2017 19:47:01 +0000 (UTC)
From: "Benjamin Lerer (JIRA)" <jira@apache.org>
To: commits@cassandra.apache.org
Message-ID: <JIRA.13025986.1481037591000.292199.1507319221084@Atlassian.JIRA>
In-Reply-To: <JIRA.13025986.1481037591000@Atlassian.JIRA>
References: <JIRA.13025986.1481037591000@Atlassian.JIRA> <JIRA.13025986.1481037591845@jira-lw-us.apache.org>
Subject: [jira] [Commented] (CASSANDRA-13006) Disable automatic heap dumps
 on OOM error
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
archived-at: Fri, 06 Oct 2017 19:47:09 -0000


    [ https://issues.apache.org/jira/browse/CASSANDRA-13006?page=3Dcom.atla=
ssian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=
=3D16195149#comment-16195149 ]=20

Benjamin Lerer commented on CASSANDRA-13006:
--------------------------------------------


[~urandom], [~brandon.williams], [~tjake]

Sorry, for the delay.

I pushed some patches for [2.2|https://github.com/apache/cassandra/compare/=
cassandra-2.2...blerer:13006-2.2], [3.0|https://github.com/apache/cassandra=
/compare/cassandra-3.0...blerer:13006-3.0],  [3.11|https://github.com/apach=
e/cassandra/compare/cassandra-3.11...blerer:13006-3.11] and [trunk|https://=
github.com/apache/cassandra/compare/trunk...blerer:13006-trunk].

The branches differs only at the level of the configuration files ({{cassan=
dra-env.sh}} and {{cassandra-env.ps1}}).=20

The patches let the JVM handle the {{OutOfMemoryErrors}} throught the JVM {=
{OnOutOfMemoryError}}, {{ExitOnOutOfMemoryError}} or {{CrashOnOutOfMemoryEr=
ror}} options.
As the {{ExitOnOutOfMemoryError}} and {{CrashOnOutOfMemoryError}} options a=
re only supported since Oracle JDK 7 update 101 and since JDK 8 update 92, =
Cassandra uses by default the {{OnOutOfMemoryError}} option.

A startup check will emit a warning if none of the options is used. This ch=
eck is there to ensure that {{OOM}} errors are properly handled and that C*=
 cannot continue to run in an unstable state that could cause data corrupti=
on.
The patch add no check for the {{HeapDumpOnOutOfMemoryError}} option as in =
some cases administrators prefer to disable them.

The {{cassandra-env.sh}} has a new variable {{JVM_ON_OUT_OF_MEMORY_ERROR_OP=
T}} which should be use to specify the {{OnOutOfMemoryError}} option. As ba=
sh commands split words on white spaces without taking quotes into account,=
 specifying the {{OnOutOfMemoryError}} as part of the {{JVM_OPTS}} variable=
 cannot work for an option value such as: {{kill -9 %p}}.

Before generating an heap dump, C* use to log an Heap histogram using {{jma=
p}}. If the heap size was large, reading the heap dump could take a few hou=
rs and the heap histogram can help to debug the problem much faster. The pa=
tches keep the posibility to print an heap histogram on OOM error but disab=
les it by default. To enable it the {{cassandra.printHeapHistogramOnOutOfMe=
moryError}} system property must be set to {{true}}.
As generating the histogram for only the live objects (using {{jmap histo:l=
ive}}) would trigger a garbage collection before generating the histogram, =
I prefered to stick with {{jmap histo}} to minimize the risks. =20

The previous implementation was suffering of 2 problems:
* If several OOM errors were thrown in a short time span, each of them woul=
d trigger an heap histogram and an heap dump (see this [comment|https://iss=
ues.apache.org/jira/browse/CASSANDRA-13006?focusedCommentId=3D16118421&page=
=3Dcom.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment=
-16118421])
* If an exception was thrown while C* was trying to generate the heap dump,=
 C* would not be shutdown and would continue running in an unstable state (=
see CASSANDRA-13886)

The patches fix those problems for the case were an heap histogram need to =
be logged. In the case were the histogram is not requested those problems d=
o not exist anymore.

CI looks good for the unit tests.=20
The changes to the {{cassandra}} startup script break the DTests but before=
 changing the {{DTests}} framework I would prefer having a first review of =
the patches.

[~JoshuaMcKenzie] could you review the patches?=20

> Disable automatic heap dumps on OOM error
> -----------------------------------------
>
>                 Key: CASSANDRA-13006
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-1300=
6
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Configuration
>            Reporter: anmols
>            Assignee: Benjamin Lerer
>            Priority: Minor
>             Fix For: 3.0.15
>
>         Attachments: 13006-3.0.9.txt
>
>
> With=C2=A0CASSANDRA-9861, a change was added to enable collecting heap du=
mps by default if the process encountered an OOM error. These heap dumps ar=
e stored in the Apache Cassandra home directory unless configured otherwise=
 (see=C2=A0[Cassandra Support Document|https://support.datastax.com/hc/en-u=
s/articles/204225959-Generating-and-Analyzing-Heap-Dumps] for this feature)=
.
> =C2=A0
> The creation and storage of heap dumps aides debugging and investigative =
workflows, but is not be desirable for a production environment where these=
 heap dumps may occupy a large amount of disk space and require manual inte=
rvention for cleanups.=C2=A0
> =C2=A0
> Managing heap dumps on out of memory errors and configuring the paths for=
 these heap dumps are available as JVM options in JVM. The current behavior=
 conflicts with the Boolean JVM flag HeapDumpOnOutOfMemoryError.=C2=A0
> =C2=A0
> A patch can be proposed here that would make the heap dump on OOM error h=
onor the=C2=A0HeapDumpOnOutOfMemoryError=C2=A0flag. Users who would want to=
 still generate heap dumps on OOM errors can set the=C2=A0-XX:+HeapDumpOnOu=
tOfMemoryError=C2=A0JVM option.


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org