Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id E683310CBA for ; Wed, 4 Sep 2013 14:15:01 +0000 (UTC) Received: (qmail 25015 invoked by uid 500); 4 Sep 2013 14:14:59 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 24956 invoked by uid 500); 4 Sep 2013 14:14:58 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 24928 invoked by uid 99); 4 Sep 2013 14:14:57 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Sep 2013 14:14:57 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [91.209.25.138] (HELO cer69mx21.cirtil.fr) (91.209.25.138) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Sep 2013 14:14:51 +0000 X-PJ: In-Reply-To: <6C63C1AE-4F0E-4B24-9F66-89C53FEEFDC6@nordsc.com> References: <6C63C1AE-4F0E-4B24-9F66-89C53FEEFDC6@nordsc.com> To: user@cassandra.apache.org MIME-Version: 1.0 Subject: Re: Cassandra shuts down; was:Cassandra crashes X-KeepSent: FA4B3199:CA772FCC-C1257BDC:004E2272; type=4; name=$KeepSent From: Romain HARDOUIN Message-ID: Date: Wed, 4 Sep 2013 16:14:28 +0200 Content-Type: multipart/alternative; boundary="=_alternative 004E3AAEC1257BDC_=" X-Virus-Checked: Checked by ClamAV on apache.org Message en plusieurs parties au format MIME --=_alternative 004E3AAEC1257BDC_= Content-Type: text/plain; charset="US-ASCII" Have you tried to tweak settings like memtable_total_space_in_mb and flush_largest_memtables_at? Also, the compaction manager seems to be pretty busy, take a look at in_memory_compaction_limit_in_mb. And with SSD hardware you should modifiy multithreaded_compaction, compaction_throughput_mb_per_sec, concurrent_reads and concurrent_writes. Of course 2GB of RAM is low but tweak these settings might help you. Maybe some guru could confirm/infirm that. De : Jan Algermissen A : user@cassandra.apache.org, Date : 04/09/2013 12:29 Objet : Re: Cassandra shuts down; was:Cassandra crashes Romain, On 04.09.2013, at 11:11, Romain HARDOUIN wrote: > Maybe you should include the end of Cassandra logs. There is nothing that seems interesting in cassandra.log. Below you find system.log. > What comes to my mind when I read your first post is OOM killer. > But what you describe later is not the case. > Just to be sure, have you checked /var/log/messages? Nothing there, just occasional Firewall TCP rejections. Somehow I think I am simply overloading the whole cluster (see the hinted handoff messages in the log). Could that be due to the limited memory of 2GB my nodes have? IOW, not enough space to buffer up the writes before dumping to disk? Also, my overall write performance is actually pretty bad compared to what I read about C*. Before I thought it was the client doing to much work or the network. Turns out that's not the case. I'd expect C* to sort of just suck in my rather small amount of data - must be me, not using the right configuration. Oh well, I'll get there :-) Thanks anyhow. Jan --=_alternative 004E3AAEC1257BDC_= Content-Type: text/html; charset="US-ASCII" Have you tried to tweak settings like memtable_total_space_in_mb and flush_largest_memtables_at?
Also, the compaction manager seems to be pretty busy, take a look at in_memory_compaction_limit_in_mb.
And with SSD hardware you should modifiy multithreaded_compaction, compaction_throughput_mb_per_sec, concurrent_reads and concurrent_writes.
Of course 2GB of RAM is low but tweak these settings might help you.
Maybe some guru could confirm/infirm that.




De :        Jan Algermissen <jan.algermissen@nordsc.com>
A :        user@cassandra.apache.org,
Date :        04/09/2013 12:29
Objet :        Re: Cassandra shuts down; was:Cassandra crashes



Romain,


On 04.09.2013, at 11:11, Romain HARDOUIN <romain.hardouin@urssaf.fr> wrote:

> Maybe you should include the end of Cassandra logs.

There is nothing that seems interesting in cassandra.log. Below you find system.log.

> What comes to my mind when I read your first post is OOM killer.
> But what you describe later is not the case.
> Just to be sure, have you checked /var/log/messages?

Nothing there, just occasional Firewall TCP rejections.

Somehow I think I am simply overloading the whole cluster (see the hinted handoff messages in the log). Could that be due to the limited memory of 2GB my nodes have? IOW, not enough space to buffer up the writes before dumping to disk?

Also, my overall write performance is actually pretty bad compared to what I read about C*. Before I thought it was the client doing to much work or the network. Turns out that's not the case.

I'd expect C* to sort of just suck in my rather small amount of data - must be me, not using the right configuration. Oh well, I'll get there :-) Thanks anyhow.

Jan
--=_alternative 004E3AAEC1257BDC_=--