From user-return-37611-apmail-cassandra-user-archive=cassandra.apache.org@cassandra.apache.org Tue Nov 19 21:00:19 2013 Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id A1DB110D01 for ; Tue, 19 Nov 2013 21:00:19 +0000 (UTC) Received: (qmail 67282 invoked by uid 500); 19 Nov 2013 21:00:17 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 67240 invoked by uid 500); 19 Nov 2013 21:00:16 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 67232 invoked by uid 99); 19 Nov 2013 21:00:16 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 19 Nov 2013 21:00:16 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy includes SPF record at spf.trusted-forwarder.org) Received: from [209.85.220.45] (HELO mail-pa0-f45.google.com) (209.85.220.45) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 19 Nov 2013 21:00:09 +0000 Received: by mail-pa0-f45.google.com with SMTP id kp14so3100507pab.18 for ; Tue, 19 Nov 2013 12:59:48 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:content-type:message-id:mime-version :subject:date:references:to:in-reply-to; bh=VLkmKkM9VX8YLmSev2XOpVAmsju/ZENZp34ExEbLAdY=; b=UgceRkXPf3bZcHzCpfK4exuXMFtMx4fqT1IM+uk7EDEeHA28rw5NskGBb/oytJnzoV T+3NDDk/eS7wl1ynWs+5LAeZFsFZJ2gkD38fcOCV/k6AapV5NeSmdrdvjGg2UPk0m6qp PZYQWJOVCCXzYzunXpAMuZ8iZ6Krxon7xjButdu8ZgtC73tofkl2W6pCMvBKlPijE8si ZwuyzNm1Icg7PKhpRy4GC+F9ThFj0gQ5OMPIjmZGiOmDxriYfDTfxNACcl4zRyGp+fzb w1hrvMgucbsvhzpum1Y6Xbu2WuQ2BhqORsCWrYEqIEYruxQ13lvYKOs0/+wKGfJhwFW7 er8g== X-Gm-Message-State: ALoCoQkm+dLxNjThls63BDpVqTdQbmffCQX6BNFJtb7xlE4PG+WSzAP5oC/mkWoQWDVZtmppGYNd X-Received: by 10.68.103.163 with SMTP id fx3mr20717104pbb.59.1384894788769; Tue, 19 Nov 2013 12:59:48 -0800 (PST) Received: from [172.16.1.20] ([203.86.207.101]) by mx.google.com with ESMTPSA id i10sm36974855pat.11.2013.11.19.12.59.47 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 19 Nov 2013 12:59:48 -0800 (PST) From: Aaron Morton Content-Type: multipart/alternative; boundary="Apple-Mail=_43C19E89-F5C0-496D-BB03-8F7277DDDB5C" Message-Id: <503C3406-26F1-4731-9DC9-3D72127F85DC@thelastpickle.com> Mime-Version: 1.0 (Mac OS X Mail 7.0 \(1822\)) Subject: Re: making sense of output from Eclipse Memory Analyzer tool taken from .hprof file Date: Wed, 20 Nov 2013 09:59:44 +1300 References: <9FD742B0-27D6-4333-A289-18E9A25FCA98@thelastpickle.com> <52855DFA.6010508@gmail.com> To: Cassandra User In-Reply-To: <52855DFA.6010508@gmail.com> X-Mailer: Apple Mail (2.1822) X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail=_43C19E89-F5C0-496D-BB03-8F7277DDDB5C Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=windows-1252 What version of cassandra are you using ? What are the JVM settings? (check with ps aux | grep cassandra) OOM in cassandra 1.2+ is rare but there is also = https://issues.apache.org/jira/browse/CASSANDRA-5706 and = https://issues.apache.org/jira/browse/CASSANDRA-6087 > One instance of "org.apache.cassandra.db.ColumnFamilyStore" loaded by = "sun.misc.Launcher$AppClassLoader @ 0x613e1bdc8" occupies 984,094,664 = (11.64%) bytes. 938MB is a bit of memory, the CFS and data tracker are dealing with the = memtable. This may indicate things are not being flushed from memory = correctly.=20 > =95java.lang.Thread @ 0x73e1f74c8 CompactionExecutor:158 - 839,225,000 = (9.92%) bytes. > =95java.lang.Thread @ 0x717f08178 MutationStage:31 - 809,909,192 = (9.58%) bytes. > =95java.lang.Thread @ 0x717f082c8 MutationStage:5 - 649,667,472 = (7.68%) bytes. > =95java.lang.Thread @ 0x717f083a8 MutationStage:21 - 498,081,544 = (5.89%) bytes. > =95java.lang.Thread @ 0x71b357e70 MutationStage:11 - 444,931,288 = (5.26%) bytes. maybe very big rows and/or very big mutations.=20 hope that helps.=20 ----------------- Aaron Morton New Zealand @aaronmorton Co-Founder & Principal Consultant Apache Cassandra Consulting http://www.thelastpickle.com On 15/11/2013, at 12:34 pm, Mike Koh wrote: > I am investigating Java Out of memory heap errors. So I created an = .hprof file and loaded it into Eclipse Memory Analyzer Tool which gave = some "Problem Suspects". >=20 > First one looks like: > ---- > One instance of "org.apache.cassandra.db.ColumnFamilyStore" loaded by = "sun.misc.Launcher$AppClassLoader @ 0x613e1bdc8" occupies 984,094,664 = (11.64%) bytes. The memory is accumulated in one instance of = "org.apache.cassandra.db.DataTracker$View" loaded by = "sun.misc.Launcher$AppClassLoader @ 0x613e1bdc8". > ---- >=20 > If I click around into the verbiage, I believe I can pick out the name = of a column family but that is about it. Can someone explain what the = above means in more detail and if it is indicative of a problem? >=20 >=20 > Next one looks like: > ----- > =95java.lang.Thread @ 0x73e1f74c8 CompactionExecutor:158 - 839,225,000 = (9.92%) bytes. > =95java.lang.Thread @ 0x717f08178 MutationStage:31 - 809,909,192 = (9.58%) bytes. > =95java.lang.Thread @ 0x717f082c8 MutationStage:5 - 649,667,472 = (7.68%) bytes. > =95java.lang.Thread @ 0x717f083a8 MutationStage:21 - 498,081,544 = (5.89%) bytes. > =95java.lang.Thread @ 0x71b357e70 MutationStage:11 - 444,931,288 = (5.26%) bytes. > ------ > If I click into the verbiage, they above Compaction and Mutations all = seem to be referencing the same column family. Are the above related? Is = there a way I can tell more exactly what is being compacted and/or = mutated more specifically than which column family? --Apple-Mail=_43C19E89-F5C0-496D-BB03-8F7277DDDB5C Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=windows-1252 What = version of cassandra are you using ?
What are the JVM settings? = (check with ps aux | grep = cassandra)



One instance of = "org.apache.cassandra.db.ColumnFamilyStore" loaded by = "sun.misc.Launcher$AppClassLoader @ 0x613e1bdc8" occupies 984,094,664 = (11.64%) bytes.938MB is a bit of memory, the CFS and data = tracker are dealing with the memtable. This may indicate things are not = being flushed from memory = correctly. 

=95java.lang.Thread @ 0x73e1f74c8 CompactionExecutor:158 - = 839,225,000 (9.92%) bytes.
=95java.lang.Thread @ 0x717f08178 = MutationStage:31 - 809,909,192 (9.58%) bytes.
=95java.lang.Thread @ = 0x717f082c8 MutationStage:5 - 649,667,472 (7.68%) = bytes.
=95java.lang.Thread @ 0x717f083a8 MutationStage:21 - = 498,081,544 (5.89%) bytes.
=95java.lang.Thread @ 0x71b357e70 = MutationStage:11 - 444,931,288 (5.26%) bytes.
maybe very big = rows and/or very big mutations. 

hope that = helps. 

http://www.thelastpickle.com

On 15/11/2013, at 12:34 pm, Mike Koh <defmikekoh@gmail.com> = wrote:

I am investigating Java Out of memory heap errors. So I = created an .hprof file and loaded it into Eclipse Memory Analyzer Tool = which gave some "Problem Suspects".

First one looks = like:
----
One instance of = "org.apache.cassandra.db.ColumnFamilyStore" loaded by = "sun.misc.Launcher$AppClassLoader @ 0x613e1bdc8" occupies 984,094,664 = (11.64%) bytes. The memory is accumulated in one instance of = "org.apache.cassandra.db.DataTracker$View" loaded by = "sun.misc.Launcher$AppClassLoader @ 0x613e1bdc8".
----

If I = click around into the verbiage, I believe I can pick out the name of a = column family but that is about it. Can someone explain what the above = means in more detail and if it is indicative of a = problem?


Next one looks like:
-----
=95java.lang.Thread = @ 0x73e1f74c8 CompactionExecutor:158 - 839,225,000 (9.92%) = bytes.
=95java.lang.Thread @ 0x717f08178 MutationStage:31 - = 809,909,192 (9.58%) bytes.
=95java.lang.Thread @ 0x717f082c8 = MutationStage:5 - 649,667,472 (7.68%) bytes.
=95java.lang.Thread @ = 0x717f083a8 MutationStage:21 - 498,081,544 (5.89%) = bytes.
=95java.lang.Thread @ 0x71b357e70 MutationStage:11 - = 444,931,288 (5.26%) bytes.
------
If I click into the verbiage, = they above Compaction and Mutations all seem to be referencing the same = column family. Are the above related? Is there a way I can tell more = exactly what is being compacted and/or mutated more specifically than = which column = family?

= --Apple-Mail=_43C19E89-F5C0-496D-BB03-8F7277DDDB5C--