Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id D6AE710335 for ; Wed, 18 Sep 2013 13:04:20 +0000 (UTC) Received: (qmail 22345 invoked by uid 500); 18 Sep 2013 13:04:15 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 22327 invoked by uid 500); 18 Sep 2013 13:04:14 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 22267 invoked by uid 99); 18 Sep 2013 13:04:11 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Sep 2013 13:04:11 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_REMOTE_IMAGE X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of ken.hancock@schange.com designates 74.125.82.51 as permitted sender) Received: from [74.125.82.51] (HELO mail-wg0-f51.google.com) (74.125.82.51) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Sep 2013 13:04:03 +0000 Received: by mail-wg0-f51.google.com with SMTP id c11so6311870wgh.6 for ; Wed, 18 Sep 2013 06:03:42 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:content-type; bh=2zl2240gJBlpqs3+Eq8tRYxXF0KWa2DVmy+LhE5skS8=; b=mSwCidOy06JpNjlmdioQBUmtXdMSCAOUv1cq/HPpBN9j+MHOabE2GP24B7iF61seoC ZTN3HeGf8WTe4x5uZqZx9W5E8aNRkC46nJU+FthJuJLiPGsG1Jm2P3MNN9RVLjfaYl1h fbqJrRhjLEpswovaVtC05uC1vHIVRcFOU6R8GOd0E6zd+/gxHGJ+QVn0Hn46H40O3zAx lGPp4Nm4TpasGatqHP85a5njp+XDk1SKuN8MK3br9Rhk/PgHWrOkZOQCqp3Ni5YuhH2I jIk1z5tf9iTlvfCDFyDT8CmhSQnA5oFGX4ZHfnxQ5JXank9wY6D9qccOQZxr1ZNus6W7 K9Uw== X-Gm-Message-State: ALoCoQn+Z8W14ZLeddLZsD4o2DMGKyK4tsWLoga+K1H4iYxfWrKrmk7myb5C7dxJruXj5mX4XUmh X-Received: by 10.180.211.206 with SMTP id ne14mr7113223wic.30.1379509422311; Wed, 18 Sep 2013 06:03:42 -0700 (PDT) MIME-Version: 1.0 Received: by 10.194.46.198 with HTTP; Wed, 18 Sep 2013 06:03:22 -0700 (PDT) In-Reply-To: References: From: Ken Hancock Date: Wed, 18 Sep 2013 09:03:22 -0400 Message-ID: Subject: Re: cassandra just gone..no heap dump, no log info To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=001a11c37d4414dd8e04e6a8113b X-Virus-Checked: Checked by ClamAV on apache.org --001a11c37d4414dd8e04e6a8113b Content-Type: text/plain; charset=ISO-8859-1 We ran into this while tuning heap sizes. With Cassandra 1.2 making use of off-heap memory, if we made our JVM too large relative to the server memory, the system would just bail. We found for our app that the limit of the JVM size relative to server memory was about 50%. On Wed, Sep 18, 2013 at 8:57 AM, Juan Manuel Formoso wrote: > This shouldn't happen if you have swap active in the server > > On Wednesday, September 18, 2013, Franc Carter wrote: > >> >> A random guess - possibly an OOM (Out of Memory) where Linux will kill a >> process to recover memory when it is desperately low on memory. Have a look >> in either your syslog output of the output of dmesg >> >> cheers >> >> >> On Wed, Sep 18, 2013 at 10:21 PM, Hiller, Dean wrote: >> >>> Anyone know how to debug cassandra processes just exiting? There is no >>> info in the cassandra logs and there is no heap dump file(which in the past >>> has shown up in /opt/cassandra/bin directory for me). >>> >>> This occurs when running a map/reduce job that put severe load on the >>> system. The logs look completely fine. I find it odd >>> >>> 1. No logs of why it exited at all >>> 2. No heap dump which would imply there would be no logs as it crashed >>> >>> Is there any other way a process can die and linux would log it somehow? >>> (like running out of memory) >>> >>> Thanks, >>> Dean >>> >> >> >> >> -- >> >> *Franc Carter* | Systems architect | Sirca Ltd >> >> franc.carter@sirca.org.au | www.sirca.org.au >> >> Tel: +61 2 8355 2514 >> >> Level 4, 55 Harrington St, The Rocks NSW 2000 >> >> PO Box H58, Australia Square, Sydney NSW 1215 >> >> >> > > -- > *Juan Manuel Formoso > *Senior Geek > http://twitter.com/juanformoso > http://seniorgeek.com.ar > LLAP > -- *Ken Hancock *| System Architect, Advanced Advertising SeaChange International 50 Nagog Park Acton, Massachusetts 01720 ken.hancock@schange.com | www.schange.com | NASDAQ:SEAC Office: +1 (978) 889-3329 | [image: Google Talk:] ken.hancock@schange.com | [image: Skype:]hancockks | [image: Yahoo IM:]hancockks [image: LinkedIn] [image: SeaChange International] This e-mail and any attachments may contain information which is SeaChange International confidential. The information enclosed is intended only for the addressees herein and may not be copied or forwarded without permission from SeaChange International. --001a11c37d4414dd8e04e6a8113b Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
We ran into this while tuning heap sizes.=A0 With Cassandr= a 1.2 making use of off-heap memory, if we made our JVM too large relative = to the server memory, the system would just bail.=A0 We found for our app t= hat the limit of the JVM size relative to server memory was about 50%.


On Wed,= Sep 18, 2013 at 8:57 AM, Juan Manuel Formoso <jformoso@gmail.com>= wrote:
This shouldn't happen if you have swap a= ctive in the server=A0

On Wednesday, September 18, 2013= , Franc Carter wrote:

A random guess - possibly an OOM (Out = of Memory) where Linux will kill a process to recover memory when it is des= perately low on memory. Have a look in either your syslog output of the out= put of dmesg

cheers


On Wed, Sep 18, 2013 at 10:21 PM, Hiller, Dean <Dean.Hiller@nrel.gov> wrote:
Anyone know how to debug cassandra processes= just exiting? =A0There is no info in the cassandra logs and there is no he= ap dump file(which in the past has shown up in /opt/cassandra/bin directory= for me).

This occurs when running a map/reduce job that put severe load on the syste= m. =A0The logs look completely fine. =A0I find it odd

=A01. =A0No logs of why it exited at all
=A02. =A0No heap dump which would imply there would be no logs as it crashe= d

Is there any other way a process can die and linux would log it somehow? = =A0(like running out of memory)

Thanks,
Dean



--

Franc Carter<= /b> |<= /span> Systems architect | Sirca Ltd

franc.carter@sirca.org.au=A0|=A0= www.sirca.org.au

Tel:= =A0+61 2 8355 2514

Level 4, 55 Harrington St, The Rocks NSW 2000

PO Box H58, Australia Square, Sydney NSW 1215<= /span>


<= br>
--
Juan Manuel Formoso
Senior Geek
http://twitter.com/juanformoso
http://seniorgeek.com.ar
LLAP=



--
Ken Hancock=A0| System Archi= tect, Advanced Advertising=A0
SeaChange International= =A0
50 Nagog Park
Acton, Massachusetts 01720
ken.hancock@schange.com= =A0|=A0www.schange.co= m=A0| NASDAQ:SEAC=A0
Office: +1 (978) 889-3329=A0|=A03D"Google=A0ken.hancock@schange.com=A0|=A03D"Skyp=hancockks=A0|=A03D"Yahoohancockks		 3D"LinkedIn"

3D"SeaChange
This e-mail and any attachments may contain information which is SeaChange= International confidential. The information enclosed is intended only for = the addressees herein and may not be copied or forwarded without permission= from SeaChange International.
--001a11c37d4414dd8e04e6a8113b--