Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id BA945EFF4 for ; Thu, 21 Feb 2013 17:31:49 +0000 (UTC) Received: (qmail 86768 invoked by uid 500); 21 Feb 2013 17:31:47 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 86711 invoked by uid 500); 21 Feb 2013 17:31:47 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 86702 invoked by uid 99); 21 Feb 2013 17:31:47 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Feb 2013 17:31:47 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [208.113.200.5] (HELO homiemail-a81.g.dreamhost.com) (208.113.200.5) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Feb 2013 17:31:39 +0000 Received: from homiemail-a81.g.dreamhost.com (localhost [127.0.0.1]) by homiemail-a81.g.dreamhost.com (Postfix) with ESMTP id 8A49BA8090 for ; Thu, 21 Feb 2013 09:31:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=thelastpickle.com; h=from :content-type:message-id:mime-version:subject:date:references:to :in-reply-to; s=thelastpickle.com; bh=+fLcizRCF9J7uoYw2i6TCS4F1W Q=; b=uVGqJJLoeYg6EFOEGWTXLwTcZlGJYYgQE3mK7njfwVCaDQvdGYVaajvViz J2+3K/oeQ7zEzphMN3W3OSNIzBMx7smY0XpaolWq008rDy2Da2wk4EsgqK24hg8V UH77redkAvpHgzAqBZ4TUby6JbxZxM87wwyTR6CR7XE/gVIMU= Received: from [172.16.1.8] (unknown [203.86.207.101]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: aaron@thelastpickle.com) by homiemail-a81.g.dreamhost.com (Postfix) with ESMTPSA id 0597FA8081 for ; Thu, 21 Feb 2013 09:31:17 -0800 (PST) From: aaron morton Content-Type: multipart/alternative; boundary="Apple-Mail=_54E625CB-5C56-4E03-BF73-77DFD0C39FDA" Message-Id: <98C95F6D-9BAB-439F-9EE6-1EAA7A6D8A85@thelastpickle.com> Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\)) Subject: Re: very confused by jmap dump of cassandra Date: Fri, 22 Feb 2013 06:31:18 +1300 References: To: user@cassandra.apache.org In-Reply-To: X-Mailer: Apple Mail (2.1499) X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail=_54E625CB-5C56-4E03-BF73-77DFD0C39FDA Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=windows-1252 Cannot comment too much on the jmap but I can add my general "compaction = is hurting" strategy.=20 Try any or all of the following to get to a stable setup, then increase = until things go bang.=20 Set concurrent compactors to 2.=20 Reduce compaction throughput by half.=20 Reduce in_memory_compaction_limit.=20 If you see compactions using a lot of sstables in the logs, reduce = max_compaction_threshold.=20 =20 > I can easily go higher than 8G on these systems as I have 32gig each = node, but there was docs that said 8G is better for GC.=20 More JVM memory is not the answer.=20 Cheers ----------------- Aaron Morton Freelance Cassandra Developer New Zealand @aaronmorton http://www.thelastpickle.com On 21/02/2013, at 7:49 AM, "Hiller, Dean" wrote: > I took this jmap dump of cassandra(in production). Before I restarted = the whole production cluster, I had some nodes running compaction and it = looked like all memory had been consumed(kind of like cassandra is not = clearing out the caches or memtables fast enough). I am trying to still = debug compaction causes slowness on the cluster since all cassandra.yaml = files are pretty much the defaults with size tiered compaction. >=20 > The weird thing is I dump and get a 5.4G heap.bin file and load that = into ecipse who tells me total is 142.8MB=85.what???? So low????, top = was showing 1.9G at the time(and I took this top snapshot later(2 hours = after)=85 (how is eclipse profile telling me the jmap showed 142.8MB in = use instead of 1.9G in use?) >=20 > Tasks: 398 total, 1 running, 397 sleeping, 0 stopped, 0 zombie > Cpu(s): 2.8%us, 0.5%sy, 0.0%ni, 96.5%id, 0.1%wa, 0.0%hi, 0.1%si, = 0.0%st > Mem: 32854680k total, 31910708k used, 943972k free, 89776k = buffers > Swap: 33554424k total, 18288k used, 33536136k free, 23428596k = cached >=20 > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 20909 cassandr 20 0 64.1g 9.2g 2.1g S 75.7 29.4 182:37.92 java > 22455 cassandr 20 0 15288 1340 824 R 3.9 0.0 0:00.02 top >=20 > It almost seems like cassandra is not being good about memory = management here as we slowly get into a situation where compaction is = run which takes out our memory(configured for 8G). I can easily go = higher than 8G on these systems as I have 32gig each node, but there was = docs that said 8G is better for GC. Has anyone else taken a jmap dump = of cassandra? >=20 > Thanks, > Dean >=20 --Apple-Mail=_54E625CB-5C56-4E03-BF73-77DFD0C39FDA Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=windows-1252
Try any or = all of the following to get to a stable setup, then increase until = things go bang. 

Set concurrent compactors = to 2. 
Reduce compaction throughput by = half. 
Reduce = in_memory_compaction_limit. 
If you see compactions using = a lot of sstables in the logs, reduce = max_compaction_threshold. 
 
 I can easily go higher than 8G on these systems as I = have 32gig each node, but there was docs that said 8G is better for = GC. 
More JVM memory is not the = answer. 

Cheers

http://www.thelastpickle.com

On 21/02/2013, at 7:49 AM, "Hiller, Dean" <Dean.Hiller@nrel.gov> = wrote:

I took this jmap dump of cassandra(in production). =  Before I restarted the whole production cluster, I had some nodes = running compaction and it looked like all memory had been consumed(kind = of like cassandra is not clearing out the caches or memtables fast = enough).  I am trying to still debug compaction causes slowness on = the cluster since all cassandra.yaml files are pretty much the defaults = with size tiered compaction.

The weird thing is I dump and get a = 5.4G heap.bin file and load that into ecipse who tells me total is = 142.8MB=85.what???? So low????, top was showing 1.9G at the time(and I = took this top snapshot later(2 hours after)=85 (how is eclipse profile = telling me the jmap showed 142.8MB in use instead of 1.9G in = use?)

Tasks: 398 total,   1 running, 397 sleeping, =   0 stopped,   0 zombie
Cpu(s):  2.8%us, =  0.5%sy,  0.0%ni, 96.5%id,  0.1%wa,  0.0%hi, =  0.1%si,  0.0%st
Mem:  32854680k total, 31910708k = used,   943972k free,    89776k = buffers
Swap: 33554424k total,    18288k used, = 33536136k free, 23428596k cached

 PID USER =      PR  NI  VIRT  RES  SHR = S %CPU %MEM    TIME+  COMMAND
20909 cassandr =  20   0 64.1g 9.2g 2.1g S 75.7 29.4 182:37.92 = java
22455 cassandr  20   0 15288 1340  824 R =  3.9  0.0   0:00.02 top

It almost seems like = cassandra is not being good about memory management here as we slowly = get into a situation where compaction is run which takes out our = memory(configured for 8G).  I can easily go higher than 8G on these = systems as I have 32gig each node, but there was docs that said 8G is = better for GC.  Has anyone else taken a jmap dump of = cassandra?

Thanks,
Dean


= --Apple-Mail=_54E625CB-5C56-4E03-BF73-77DFD0C39FDA--