incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hiller, Dean" <Dean.Hil...@nrel.gov>
Subject Re: cassandra just gone..no heap dump, no log info
Date Wed, 18 Sep 2013 13:11:21 GMT
We had hs_err_pid files moths ago and it was happening every 6 days or so and we switched to
this JVM and we have not seen one since(including today)…that worked for us at least.

java version "1.6.0_41"
Java(TM) SE Runtime Environment (build 1.6.0_41-b02)
Java HotSpot(TM) 64-Bit Server VM (build 20.14-b01, mixed mode)

Dean

From: java8964 java8964 <java8964@hotmail.com<mailto:java8964@hotmail.com>>
Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org>>
Date: Wednesday, September 18, 2013 7:08 AM
To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org>>
Subject: RE: cassandra just gone..no heap dump, no log info

We faced same issue sometimes too.

1) Linux OOM killer kill your Cassandra process. You should find this event log in /var/log/message.
2) The JVM crashed. You should be able to find the hs_err_pid file under /tmp folder, if you
didn't specify the location when you started your JVM.

We still cannot find out the root reason why the JVM crashed. The core dump file indicates
the JVM crashed in compile Java code, several times though. It didn't help even we upgraded
from latest JDK 1.6 to latest JDK 1.7.

Yong

> From: Dean.Hiller@nrel.gov<mailto:Dean.Hiller@nrel.gov>
> To: user@cassandra.apache.org<mailto:user@cassandra.apache.org>
> Date: Wed, 18 Sep 2013 06:59:07 -0600
> Subject: Re: cassandra just gone..no heap dump, no log info
>
> Swappiness is set to 60 though cassandra recommendations are to turn swap completely
off(we still have not done that as far as I know though) and sure enough linux killed it.
>
> Dean
>
> From: Juan Manuel Formoso <jformoso@gmail.com<mailto:jformoso@gmail.com><mailto:jformoso@gmail.com>>
> Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org><mailto:user@cassandra.apache.org>"
<user@cassandra.apache.org<mailto:user@cassandra.apache.org><mailto:user@cassandra.apache.org>>
> Date: Wednesday, September 18, 2013 6:57 AM
> To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org><mailto:user@cassandra.apache.org>"
<user@cassandra.apache.org<mailto:user@cassandra.apache.org><mailto:user@cassandra.apache.org>>
> Subject: Re: cassandra just gone..no heap dump, no log info
>
> This shouldn't happen if you have swap active in the server
>
> On Wednesday, September 18, 2013, Franc Carter wrote:
>
> A random guess - possibly an OOM (Out of Memory) where Linux will kill a process to recover
memory when it is desperately low on memory. Have a look in either your syslog output of the
output of dmesg
>
> cheers
>
>
> On Wed, Sep 18, 2013 at 10:21 PM, Hiller, Dean <Dean.Hiller@nrel.gov<mailto:Dean.Hiller@nrel.gov><javascript:_e({},%20'cvml',%20'Dean.Hiller@nrel.gov<mailto:%20'Dean.Hiller@nrel.gov>');>>
wrote:
> Anyone know how to debug cassandra processes just exiting? There is no info in the cassandra
logs and there is no heap dump file(which in the past has shown up in /opt/cassandra/bin directory
for me).
>
> This occurs when running a map/reduce job that put severe load on the system. The logs
look completely fine. I find it odd
>
> 1. No logs of why it exited at all
> 2. No heap dump which would imply there would be no logs as it crashed
>
> Is there any other way a process can die and linux would log it somehow? (like running
out of memory)
>
> Thanks,
> Dean
>
>
>
> --
>
> Franc Carter| Systems architect|Sirca Ltd
> <javascript:_e({},%20'cvml',%20'marc.zianideferranti@sirca.org.au<mailto:%20'marc.zianideferranti@sirca.org.au>');>
>
> franc.carter@sirca.org.au<mailto:franc.carter@sirca.org.au><javascript:_e({},%20'cvml',%20'franc.carter@sirca.org.au<mailto:%20'franc.carter@sirca.org.au>');>
| www.sirca.org.au<http://www.sirca.org.au/>
>
> Tel: +61 2 8355 2514
>
> Level 4, 55 Harrington St, The Rocks NSW 2000
>
> PO Box H58, Australia Square, Sydney NSW 1215
>
>
>
> --
> Juan Manuel Formoso
> Senior Geek
> http://twitter.com/juanformoso
> http://seniorgeek.com.ar<http://seniorgeek.com.ar/>
> LLAP

Mime
View raw message