From user-return-31582-apmail-cassandra-user-archive=cassandra.apache.org@cassandra.apache.org Fri Feb 1 17:55:39 2013 Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 9BAF3E396 for ; Fri, 1 Feb 2013 17:55:39 +0000 (UTC) Received: (qmail 60907 invoked by uid 500); 1 Feb 2013 17:55:37 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 60854 invoked by uid 500); 1 Feb 2013 17:55:37 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 60841 invoked by uid 99); 1 Feb 2013 17:55:37 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 Feb 2013 17:55:37 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [208.113.200.5] (HELO homiemail-a44.g.dreamhost.com) (208.113.200.5) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 Feb 2013 17:55:30 +0000 Received: from homiemail-a44.g.dreamhost.com (localhost [127.0.0.1]) by homiemail-a44.g.dreamhost.com (Postfix) with ESMTP id 0778211806C for ; Fri, 1 Feb 2013 09:55:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=thelastpickle.com; h=from :content-type:message-id:mime-version:subject:date:references:to :in-reply-to; s=thelastpickle.com; bh=xs5cVZESlpQoMdSoCslS/xldrM 0=; b=Waofu+mzVbPtEtbxeLI94ojOOlE8KY6KRtbct4bdPmovlkyvXz5hePkN8P Dl+RqDPWY8pJVbliQvh/74fW3im748e/5eZDul6flfl/UdsWIJeiAXMoe4kCwv5e 6E1kTQ+mcRcHxLRZ+JRygnyd6ZByLT5JSG+f1yHapmjJi7iqo= Received: from [172.16.1.8] (unknown [203.86.207.101]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: aaron@thelastpickle.com) by homiemail-a44.g.dreamhost.com (Postfix) with ESMTPSA id 6A768118060 for ; Fri, 1 Feb 2013 09:55:08 -0800 (PST) From: aaron morton Content-Type: multipart/alternative; boundary="Apple-Mail=_07819255-07A9-4F64-8B21-5A52D7E21AD4" Message-Id: Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\)) Subject: Re: CPU hotspot at BloomFilterSerializer#deserialize Date: Sat, 2 Feb 2013 06:55:07 +1300 References: To: user@cassandra.apache.org In-Reply-To: X-Mailer: Apple Mail (2.1499) X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail=_07819255-07A9-4F64-8B21-5A52D7E21AD4 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=windows-1252 > 5. the problematic Data file contains only 5 to 10 keys data but = large(2.4G) So very large rows ?=20 What does nodetool cfstats or cfhistograms say about the row sizes ?=20 > 1. what is happening? I think this is partially large rows and partially the query pattern, = this is only by roughly correct = http://thelastpickle.com/2011/07/04/Cassandra-Query-Plans/ and my talk = here http://www.datastax.com/events/cassandrasummit2012/presentations > 3. any more info required to proceed? Do some tests with different query techniques=85 Get a single named column.=20 Get the first 10 columns using the natural column order. Get the last 10 columns using the reversed order.=20 Hope that helps.=20 ----------------- Aaron Morton Freelance Cassandra Developer New Zealand @aaronmorton http://www.thelastpickle.com On 31/01/2013, at 7:20 PM, Takenori Sato wrote: > Hi all, >=20 > We have a situation that CPU loads on some of our nodes in a cluster = has spiked occasionally since the last November, which is triggered by = requests for rows that reside on two specific sstables. >=20 > We confirmed the followings(when spiked): >=20 > version: 1.0.7(current) <- 0.8.6 <- 0.8.5 <- 0.7.8 > jdk: Oracle 1.6.0 >=20 > 1. a profiling showed that BloomFilterSerializer#deserialize was the = hotspot(70% of the total load by running threads) >=20 > * the stack trace looked like this(simplified) > 90.4% - org.apache.cassandra.db.ReadVerbHandler.doVerb > 90.4% - org.apache.cassandra.db.SliceByNamesReadCommand.getRow > ... > 90.4% - = org.apache.cassandra.db.CollationController.collectTimeOrderedData > ... > 89.5% - = org.apache.cassandra.db.columniterator.SSTableNamesIterator.read > ... > 79.9% - = org.apache.cassandra.io.sstable.IndexHelper.defreezeBloomFilter > 68.9% - = org.apache.cassandra.io.sstable.BloomFilterSerializer.deserialize > 66.7% - java.io.DataInputStream.readLong >=20 > 2. Usually, 1 should be so fast that a profiling by sampling can not = detect >=20 > 3. no pressure on Cassandra's VM heap nor on machine in overal >=20 > 4. a little I/O traffic for our 8 disks/node(up to 100tps/disk by = "iostat 1 1000") >=20 > 5. the problematic Data file contains only 5 to 10 keys data but = large(2.4G) >=20 > 6. the problematic Filter file size is only 256B(could be normal) >=20 >=20 > So now, I am trying to read the Filter file in the same way = BloomFilterSerializer#deserialize does as possible as I can, in order to = see if the file is something wrong. >=20 > Could you give me some advise on: >=20 > 1. what is happening? > 2. the best way to simulate the BloomFilterSerializer#deserialize > 3. any more info required to proceed? >=20 > Thanks, > Takenori --Apple-Mail=_07819255-07A9-4F64-8B21-5A52D7E21AD4 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=windows-1252 5. the problematic Data file contains only 5 = to 10 keys data but large(2.4G)
So very large rows = ? 
What does nodetool cfstats or cfhistograms say about = the row sizes = ? 


1. what is happening?
I think this = is partially large rows and partially the query pattern, this is only by = roughly correct http:/= /thelastpickle.com/2011/07/04/Cassandra-Query-Plans/ and my = talk here = http://www.datastax.com/events/cassandrasummit2012/presentations
=

3. any more info required = to proceed?
Do some tests with different query = techniques=85

Get a single named = column. 
Get the first 10 columns using the natural = column order.
Get the last 10 columns using the reversed = order. 

Hope that = helps. 

http://www.thelastpickle.com

On 31/01/2013, at 7:20 PM, Takenori Sato <tsato@cloudian.com> = wrote:

Hi all,

We have a situation that CPU = loads on some of our nodes in a cluster has spiked occasionally since = the last November, which is triggered by requests for rows that reside = on two specific sstables.

We confirmed the followings(when = spiked):

version: 1.0.7(current) <- 0.8.6 = <- 0.8.5 <- 0.7.8
jdk: Oracle = 1.6.0

1. a profiling showed that = BloomFilterSerializer#deserialize was the hotspot(70% of the total load = by running threads)

* the stack trace looked like = this(simplified)
90.4% - = org.apache.cassandra.db.ReadVerbHandler.doVerb
90.4% - = org.apache.cassandra.db.SliceByNamesReadCommand.getRow
...
=
90.4% - = org.apache.cassandra.db.CollationController.collectTimeOrderedData
...
89.5% - = org.apache.cassandra.db.columniterator.SSTableNamesIterator.read
...
79.9% - = org.apache.cassandra.io.sstable.IndexHelper.defreezeBloomFilter
68.9% - = org.apache.cassandra.io.sstable.BloomFilterSerializer.deserialize
66.7% - java.io.DataInputStream.readLong

2. = Usually, 1 should be so fast that a profiling by sampling can not = detect

3. no pressure on Cassandra's VM heap nor on machine = in overal

4. a little I/O traffic for our 8 = disks/node(up to 100tps/disk by "iostat 1 1000")

5. the problematic Data file contains only 5 to 10 keys data but = large(2.4G)

6. the problematic Filter file size = is only 256B(could be normal)


So = now, I am trying to read the Filter file in the same way = BloomFilterSerializer#deserialize does as possible as I can, in order to = see if the file is something wrong.

Could you give me some advise = on:

1. what is happening?
2. the best = way to simulate the BloomFilterSerializer#deserialize
3. any = more info required to proceed?

Thanks,
Takenori

= --Apple-Mail=_07819255-07A9-4F64-8B21-5A52D7E21AD4--