From user-return-29934-apmail-cassandra-user-archive=cassandra.apache.org@cassandra.apache.org Wed Nov 7 13:39:44 2012 Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id C0EAEDFCF for ; Wed, 7 Nov 2012 13:39:44 +0000 (UTC) Received: (qmail 97885 invoked by uid 500); 7 Nov 2012 13:39:42 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 97257 invoked by uid 500); 7 Nov 2012 13:39:36 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 97203 invoked by uid 99); 7 Nov 2012 13:39:35 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Nov 2012 13:39:34 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW X-Spam-Check-By: apache.org Received-SPF: unknown (nike.apache.org: error in processing during lookup of bryan@appssavvy.com) Received: from [209.85.220.172] (HELO mail-vc0-f172.google.com) (209.85.220.172) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Nov 2012 13:39:27 +0000 Received: by mail-vc0-f172.google.com with SMTP id fl11so1719013vcb.31 for ; Wed, 07 Nov 2012 05:39:05 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=from:mime-version:content-type:subject:date:in-reply-to:to :references:message-id:x-mailer:x-gm-message-state; bh=MIBbVhbUt9uXqMq69VMWd+NSBd+bGNy8D9Zjym55h3w=; b=MKRMhjv7D7nN440tz+CFqQ+lIgO288yjjGL3XmElY3sie+D+AqqIiLhhXe3zQL3fIm 48o1jesMn5MTZTiKEKrHYB1PYiEM01bM4yZrS7Z2zD7ZF0cOIMo0k72okZMx5dGnOEy9 5VX0fYsrZQxh6vtFjfNu9mBmsXwGrEj0RdtpqxOPkfDqv7nbr1g8r0u41PYCjwTlu0yt /FN2erMjTBqZQLNcLt7N3M2vwlXmjZzjcTfi0ongX/zKpyGgLgHWXIvyQzfQipPS3u96 Z1GG2tzYWL/712xhyx5NspXZWwMqC6ddtobV2fEPZ+wjZRcix61VClr2uxbIcQhW1Utn 24Jg== Received: by 10.220.240.18 with SMTP id ky18mr4045923vcb.54.1352295545605; Wed, 07 Nov 2012 05:39:05 -0800 (PST) Received: from [192.168.15.6] ([50.14.254.103]) by mx.google.com with ESMTPS id u6sm10516066vdi.20.2012.11.07.05.39.04 (version=TLSv1/SSLv3 cipher=OTHER); Wed, 07 Nov 2012 05:39:05 -0800 (PST) From: Bryan Mime-Version: 1.0 (Apple Message framework v1283) Content-Type: multipart/alternative; boundary="Apple-Mail=_A81A8D9B-86C0-4779-B9C0-FCF6032672F5" Subject: Re: Questions around the heap Date: Wed, 7 Nov 2012 08:39:03 -0500 In-Reply-To: To: user@cassandra.apache.org References: Message-Id: X-Mailer: Apple Mail (2.1283) X-Gm-Message-State: ALoCoQmszHpiI0dCer6FDNzXkJoVK/iAwIcFg3PtaiVBn/L9ZM0so4DKnWUZhYVkYtA6ZC/zWT1a X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail=_A81A8D9B-86C0-4779-B9C0-FCF6032672F5 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=windows-1252 What are your bloom filter settings on your CFs? Maybe look here: = http://www.datastax.com/docs/1.1/operations/tuning#tuning-bloomfilters On Nov 7, 2012, at 4:56 AM, Alain RODRIGUEZ wrote: > Hi, >=20 > We just had some issue in production that we finally solve upgrading = hardware and increasing the heap. >=20 > Now we have 3 xLarge servers from AWS (15G RAM, 4 cpu - 8 cores). We = add them and then removed the old ones. >=20 > With full default configuration, 0.75 threshold of 4G was being reach = continuously, so I was obliged to increase the heap to 8G: >=20 > Memtable : 2G (Manually configured) > Key cache : 0.1G (min(5% of Heap (in MB), 100MB)) > System : 1G (more or less, from datastax doc) >=20 > It should use about 3 G and it actually use between 4 and 6 G. >=20 > So here are my questions: >=20 > How can we know how the heap is being used, monitor it ? > Why have I that much memory used in the heap of my new servers ? >=20 > All configurations not specified are default from 1.1.2 Cassandra. >=20 > Here is what happen to us before, why we change our hardware, if you = have any clue on what happen we would be glad to learn and maybe come = back to our old hardware. >=20 > -------------------------------- User experience = ------------------------------------------------------------------------ >=20 > We had a Cassandra 1.1.2 2 nodes cluster with RF2 and CL.ONE (R&W) = running on 2 m1.Large aws (7.5G RAM, 2 cpu - 4 cores dedicated to = Cassandra only).=20 >=20 > Cassandra.yaml was configured with 1.1.2 default options and in = cassandra-env.sh I configured a 4G heap with a 200M "new size". >=20 > That is the heap that was supposed to be used. >=20 > Memtable : 1.4G (1/3 of the heap) > Key cache : 0.1G (min(5% of Heap (in MB), 100MB)) > System : 1G (more or less, from datastax doc) >=20 > So we are around 2.5G max in theory out of 3G usable (threshold 0.75 = of the heap before flushing memtable because of pressure) >=20 > I thought it was ok regarding Datastax documentation: >=20 > "Regardless of how much RAM your hardware has, you should keep the JVM = heap size constrained by the following formula and allow the operating = system=92s file cache to do the rest: > (memtable_total_space_in_mb) + 1GB + (cache_size_estimate)" >=20 > After adding a third node and changing the RF from 2 to 3 (to allow = using CL.QUORUM and still be able to restart a node whenever we want), = things went really bad. Even if I still don't get how any of these = operations could possibly affect the heap needed. >=20 > All the 3 nodes reached the 0.75 heap threshold (I tried to increase = it to 0.85, but it was still reached). And they never came down. So my = cluster started flushing a lot and the load increased because of = unceasing compactions. This unexpected load produced latency that broke = down our service for a while. Even with the service down, Cassandra was = unable to recover. >=20 --Apple-Mail=_A81A8D9B-86C0-4779-B9C0-FCF6032672F5 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=windows-1252 What = are your bloom filter settings on your CFs? Maybe look here: http://www.datastax.com/docs/1.1/operations/tuning#tuning-bloomfilte= rs



On Nov 7, 2012, = at 4:56 AM, Alain RODRIGUEZ wrote:

Hi,

We just had some issue in = production that we finally solve upgrading hardware and increasing the = heap.

Now we have 3 xLarge servers from AWS = (15G RAM, 4 cpu - 8 cores). We add them and then removed the old = ones.

With full default configuration, 0.75 = threshold of 4G was being reach continuously, so I was obliged to = increase the heap to 8G:

Memtable  : = 2G (Manually configured)
Key cache : 0.1G (min(5% of Heap (in MB), 100MB))
System =     : 1G     (more or less, from datastax = doc)

It should use about 3 G and it = actually use between 4 and 6 G.

So here are my questions:

How = can we know how the heap is being used, monitor it ?
Why have = I that much memory used in the heap of my new servers = ?

All configurations not specified are default from 1.1.2 = Cassandra.

Here is what happen to us before, = why we change our hardware, if you have any clue on what happen we would = be glad to learn and maybe come back to our old hardware.

-------------------------------- User = experience = ------------------------------------------------------------------------

We had a Cassandra 1.1.2 2 nodes cluster with = RF2 and CL.ONE (R&W) running on 2 m1.Large aws (7.5G RAM, 2 cpu = - 4 cores dedicated to Cassandra only). 

Cassandra.yaml was configured with 1.1.2 default = options and in cassandra-env.sh I configured a 4G heap with a 200M "new = size".

That is the heap that was supposed to be = used.

Memtable  : 1.4G (1/3 of the = heap)
Key cache : 0.1G (min(5% of Heap (in MB), = 100MB))
System     : 1G     (more or less, = from datastax doc)

So we are around 2.5G max in = theory out of 3G usable (threshold 0.75 of the heap before flushing = memtable because of pressure)

I thought it was ok regarding Datastax = documentation:

"Regardless of how much RAM your hardware has, = you should keep the JVM heap size constrained by the following formula = and allow the operating system=92s file cache to do the rest:

(memtable_total_space_in_mb) + 1GB + = (cache_size_estimate)"

After adding a third node and = changing the RF from 2 to 3 (to allow using CL.QUORUM and still be = able to restart a node whenever we want), things went really bad. Even = if I still don't get how any of these operations could possibly affect = the heap needed.

All the 3 nodes reached the 0.75 heap = threshold (I tried to increase it to 0.85, but it was still reached). = And they never came down. So my cluster started flushing a lot and the = load increased because of unceasing compactions. This unexpected = load produced latency that broke down our service for a while. Even with = the service down, Cassandra was unable to recover.


= --Apple-Mail=_A81A8D9B-86C0-4779-B9C0-FCF6032672F5--