Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 88803118F7 for ; Mon, 28 Jul 2014 14:37:01 +0000 (UTC) Received: (qmail 94234 invoked by uid 500); 28 Jul 2014 14:36:58 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 94202 invoked by uid 500); 28 Jul 2014 14:36:58 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 94192 invoked by uid 99); 28 Jul 2014 14:36:58 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Jul 2014 14:36:58 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of ruchir.jha@gmail.com designates 209.85.217.180 as permitted sender) Received: from [209.85.217.180] (HELO mail-lb0-f180.google.com) (209.85.217.180) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Jul 2014 14:36:53 +0000 Received: by mail-lb0-f180.google.com with SMTP id v6so5772810lbi.25 for ; Mon, 28 Jul 2014 07:36:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=IidsjiADcZhxuDo3q0XhSm/vujbC3kQmQyH2PivuloQ=; b=ETYlR+rvRp93MvNIpCEpDG8sfOUr4oONfpseU6x8Bn/x/iKBbksoKdpb1LUjfndJxz vH3eXxdWSzcYf73iLc4mE2ge0qEsXNZtia5lmoDx48VrdyuFBwkewto+pAdMkxtF2L4Q 657/I6sxP5wPWwXtwSH5niB8opRtO0cFhLh7rCtVVBkGZFqsgsXmJ82r8xMJhO+RU8Gi GQMyfrawqk4t/6cOqNfdO53knzscxG7yFvORdaKK1JbbT9J7FeF/irV8us0pePwkcUcY FSyO59Pxznmbml0XXb0t0ZzdIobxQbWzc5o3hqQU+VodbEWIE6K9ve2GyPlZO3JTYyOf u9Hw== MIME-Version: 1.0 X-Received: by 10.112.63.168 with SMTP id h8mr20480524lbs.69.1406558192514; Mon, 28 Jul 2014 07:36:32 -0700 (PDT) Received: by 10.114.79.233 with HTTP; Mon, 28 Jul 2014 07:36:32 -0700 (PDT) In-Reply-To: References: Date: Mon, 28 Jul 2014 10:36:32 -0400 Message-ID: Subject: Re: Full GC in cassandra From: Ruchir Jha To: "user@cassandra.apache.org" Content-Type: multipart/alternative; boundary=001a11c3edf06bbca504ff41d941 X-Virus-Checked: Checked by ClamAV on apache.org --001a11c3edf06bbca504ff41d941 Content-Type: text/plain; charset=UTF-8 Doing about 5K writes / second. Avg Data Size = 1.6 TB / node. Total Data Size = 21 TB. And this is the nodetool cfstats output for one of our busiest column families: SSTable count: 10 Space used (live): 43239294899 Space used (total): 43239419603 SSTable Compression Ratio: 0.2954468408497778 Number of Keys (estimate): 63729152 Memtable Columns Count: 1921620 Memtable Data Size: 257680020 Memtable Switch Count: 9 Read Count: 6167 Read Latency: NaN ms. Write Count: 770984 Write Latency: 0.098 ms. Pending Tasks: 0 Bloom Filter False Positives: 370 Bloom Filter False Ratio: 0.00000 Bloom Filter Space Used: 80103200 Compacted row minimum size: 180 Compacted row maximum size: 3311 Compacted row mean size: 2631 Average live cells per slice (last five minutes): 73.0 Average tombstones per slice (last five minutes): 13.0 On Mon, Jul 28, 2014 at 10:14 AM, Mark Reddy wrote: > What is your data size and number of columns in Cassandra. Do you do many > deletions? > > > On Mon, Jul 28, 2014 at 2:53 PM, Ruchir Jha wrote: > >> Really curious to know what's causing the spike in Columns and >> DeletedColums below : >> >> >> 2014-07-28T09:30:27.471-0400: 127335.928: [Full GC 127335.928: [Class >> Histogram: >> num #instances #bytes class name >> ---------------------------------------------- >> 1: 132626060 6366050880 java.nio.HeapByteBuffer >> 2: 28194918 3920045528 [B >> 3: 78124737 3749987376 >> edu.stanford.ppl.concurrent.SnapTreeMap$Node >> * 4: 67650128 2164804096 >> <2164804096> org.apache.cassandra.db.Column* >> * 5: 16315310 522089920 >> org.apache.cassandra.db.DeletedColumn* >> 6: 6818 392489608 [I >> 7: 2844374 273059904 >> edu.stanford.ppl.concurrent.CopyOnWriteManager$COWEpoch >> 8: 5727000 229080000 java.util.TreeMap$Entry >> 9: 767742 182921376 [J >> 10: 2932832 140775936 >> edu.stanford.ppl.concurrent.SnapTreeMap$RootHolder >> 11: 2844375 91020000 >> edu.stanford.ppl.concurrent.CopyOnWriteManager$Latch >> 12: 4145131 66322096 >> java.util.concurrent.atomic.AtomicReference >> 13: 437874 64072392 [C >> 14: 2660844 63860256 >> java.util.concurrent.ConcurrentSkipListMap$Node >> 15: 4920 62849864 [[B >> 16: 1632063 52226016 >> edu.stanford.ppl.concurrent.SnapTreeMap >> > > --001a11c3edf06bbca504ff41d941 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Doing about 5K writes / second. Avg Data Size =3D 1.6 TB /= node. Total Data Size =3D 21 TB.=C2=A0

And this is the = nodetool cfstats output for one of our busiest column families:
<= br>
=C2=A0 SSTable count: 10
=C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 Space used (live): 43239294899
=C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Space used (total): 43239= 419603
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 SS= Table Compression Ratio: 0.2954468408497778
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Number of Keys= (estimate): 63729152
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 Memtable Columns Count: 1921620
=C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Memtable Data Size: 257680020
<= div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Memtable Switch= Count: 9
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Read Count: 61= 67
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Read L= atency: NaN ms.
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 Write Count: 770984
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 Write Latency: 0.098 ms.
=C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Pending Tasks: 0
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Bloom Filter F= alse Positives: 370
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 Bloom Filter False Ratio: 0.00000
=C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Bloom Filter Space Used: 80103200
=
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Compacted row = minimum size: 180
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Compacted row = maximum size: 3311
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 Compacted row mean size: 2631
=C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Average live cells per slice (last five min= utes): 73.0
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 Average tombstones per slice (last five minutes): 13.0



On Mon, Jul 28, 2014 at 10:14 AM, Mark Reddy <mark.redd= y@boxever.com> wrote:
What is your data size and = number of columns in Cassandra. Do you do many deletions?


On Mon, Jul 28, 2014 at 2:53 PM, Ruchir Jha <ruchir.jha@gmail.com<= /a>> wrote:
Really curious to know what's causing t= he spike in Columns and DeletedColums below :


2014-07-28T0= 9:30:27.471-0400: 127335.928: [Full GC 127335.928: [Class Histogram:
=C2=A0num =C2=A0 =C2=A0 #instances =C2=A0 =C2=A0 =C2=A0 =C2=A0 #bytes =C2= =A0class name
----------------------------------------------
=C2=A0 =C2=A02: = =C2=A0 =C2=A0 =C2=A028194918 =C2=A0 =C2=A0 3920045528 =C2=A0[B
=C2=A0 =C2=A03: =C2=A0 = =C2=A0 =C2=A078124737 =C2=A0 =C2=A0 3749987376 =C2=A0edu.stanford.ppl.concu= rrent.SnapTreeMap$Node
=C2=A0 =C2=A0= 4: =C2=A0 =C2=A0 =C2=A067650128 =C2=A0 =C2=A0=C2=A02164804096=C2=A0=C2=A0org.a= pache.cassandra.db.Column
=C2=A0 =C2=A05: =C2=A0 =C2=A0 =C2=A016315310 =C2=A0 =C2=A0 =C2=A05220899= 20 =C2=A0org.apache.cassandra.db.DeletedColumn
=C2=A0 =C2=A06: =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A06818 =C2=A0 =C2=A0 =C2=A0392489608 =C2=A0[I
=C2=A0 =C2=A07: =C2=A0 =C2=A0 =C2=A0 2844374 =C2=A0 =C2=A0 =C2=A0273059904 = =C2=A0edu.stanford.ppl.concurrent.CopyOnWriteManager$COWEpoch
=C2=A0 =C2=A08: =C2=A0 = =C2=A0 =C2=A0 5727000 =C2=A0 =C2=A0 =C2=A0229080000 =C2=A0java.util.TreeMap= $Entry
=C2=A0 =C2=A09: =C2=A0 =C2=A0 =C2=A0 =C2=A0767742 =C2=A0 =C2=A0 =C2=A018292= 1376 =C2=A0[J
=C2=A0 10: =C2=A0 =C2=A0 =C2=A0 2932832 =C2=A0 =C2=A0 =C2=A0140775936 = =C2=A0edu.stanford.ppl.concurrent.SnapTreeMap$RootHolder
=C2=A0 11: =C2=A0 =C2=A0 =C2=A0 2844375 =C2=A0 =C2=A0 =C2=A0 91020000 =C2= =A0edu.stanford.ppl.concurrent.CopyOnWriteManager$Latch
=C2=A0 12: =C2=A0 =C2=A0 =C2= =A0 4145131 =C2=A0 =C2=A0 =C2=A0 66322096 =C2=A0java.util.concurrent.atomic= .AtomicReference
=C2=A0 13: =C2= =A0 =C2=A0 =C2=A0 =C2=A0437874 =C2=A0 =C2=A0 =C2=A0 64072392 =C2=A0[C
=
=C2=A0 14: =C2= =A0 =C2=A0 =C2=A0 2660844 =C2=A0 =C2=A0 =C2=A0 63860256 =C2=A0java.util.con= current.ConcurrentSkipListMap$Node
=C2=A0 15: =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A04920 =C2=A0 =C2=A0 =C2=A0 62849864 =C2=A0[[B=
=C2=A0 16:= =C2=A0 =C2=A0 =C2=A0 1632063 =C2=A0 =C2=A0 =C2=A0 52226016 =C2=A0edu.stanf= ord.ppl.concurrent.SnapTreeMap


--001a11c3edf06bbca504ff41d941--