Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 49E3CE2D8 for ; Mon, 7 Jan 2013 20:47:58 +0000 (UTC) Received: (qmail 84339 invoked by uid 500); 7 Jan 2013 20:47:55 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 84232 invoked by uid 500); 7 Jan 2013 20:47:55 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 84222 invoked by uid 99); 7 Jan 2013 20:47:55 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jan 2013 20:47:55 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [208.113.200.5] (HELO homiemail-a59.g.dreamhost.com) (208.113.200.5) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jan 2013 20:47:50 +0000 Received: from homiemail-a59.g.dreamhost.com (localhost [127.0.0.1]) by homiemail-a59.g.dreamhost.com (Postfix) with ESMTP id 92E1B56406C for ; Mon, 7 Jan 2013 12:47:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=thelastpickle.com; h=from :content-type:message-id:mime-version:subject:date:references:to :in-reply-to; s=thelastpickle.com; bh=fD1/8uTXR0N33AvI+favDM+j+r 0=; b=xzXT6D3j8InsmDy2lfJT14hJSSQOy9+lNcJdscdI89QU2J1j338BW4LIQJ W548j94RqJpBSP14TvqG638RNv7MhAYxcBqIP1rJKIkWtVtnMI1Fxs42VOEZXk11 vVdVbhap8Zx99JuNYcvsGKB7i+k0WiX7DQwiKHl3xinS3Hyik= Received: from [172.16.1.8] (unknown [203.86.207.101]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: aaron@thelastpickle.com) by homiemail-a59.g.dreamhost.com (Postfix) with ESMTPSA id 07D8956405C for ; Mon, 7 Jan 2013 12:47:28 -0800 (PST) From: aaron morton Content-Type: multipart/alternative; boundary="Apple-Mail=_85F6A1F8-FEF1-497E-9150-FB4AA6796345" Message-Id: <7FFBA859-590C-4FE7-8B32-255B6FEC1A4D@thelastpickle.com> Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\)) Subject: Re: puzzled why my cluster is slowing down Date: Tue, 8 Jan 2013 09:47:27 +1300 References: To: user@cassandra.apache.org In-Reply-To: X-Mailer: Apple Mail (2.1499) X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail=_85F6A1F8-FEF1-497E-9150-FB4AA6796345 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=iso-8859-1 Can you slice up the "slows down" part a little more?=20 Are you saying you are getting 4500 u-sec write latency ?=20 Are you using secondary indexes? What sort of read queries are slowing down?=20 What does the schema look like ?=20 If the simple checks like CPU, iostat and GC logging in the cassandra = log do not show anything the next thing to look at is work load. Look = for things like wide gets on wide rows, long lived rows with many = deletes.=20 Cheers ----------------- Aaron Morton Freelance Cassandra Developer New Zealand @aaronmorton http://www.thelastpickle.com On 8/01/2013, at 3:39 AM, Brian Tarbox wrote: > I have a 4 node cluster with lots JVM memory and lots of system memory = that slows down when I'm doing lots of writes. >=20 > Running DataStax charts I see my read and write latency rise from = 50-100 u-secs to 1500-4500 u-secs. This is across a 12 hour data load = during which time the applied load is high but fairly constant (500-700 = writes/sec). >=20 > I'm trying to understand the slowdown: there is no memory pressure, = I've run every option under nodetool to look for bottlenecks (tpstats, = compactionStats, etc) and see none. I'm running with keycache and have = about 98% hits. >=20 > What can I check next? > Thanks! >=20 > Brian Tarbox --Apple-Mail=_85F6A1F8-FEF1-497E-9150-FB4AA6796345 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=iso-8859-1 Can = you slice up the "slows down" part a little more? 
Are you = saying you are getting 4500 u-sec write latency = ? 

Are you using secondary = indexes?
What sort of read queries are slowing = down? 
What does the schema look like = ? 

If the simple checks like CPU, iostat = and GC logging in the cassandra log do not show anything the next thing = to look at is work load. Look for things like wide gets on wide rows, = long lived rows with many = deletes. 

Cheers

http://www.thelastpickle.com

On 8/01/2013, at 3:39 AM, Brian Tarbox <tarbox@cabotresearch.com> = wrote:

I have a 4 node cluster with lots JVM memory and lots of = system memory that slows down when I'm doing lots of = writes.

Running DataStax charts I see my read and = write latency rise from 50-100 u-secs to 1500-4500 u-secs.  This is = across a 12 hour data load during which time the applied load is high = but fairly constant (500-700 writes/sec).

I'm trying to understand the slowdown: there is no = memory pressure, I've run every option under nodetool to look for = bottlenecks (tpstats, compactionStats, etc) and see none.  I'm = running with keycache and have about 98% hits.

What can I check = next?
Thanks!

Brian Tarbox

= --Apple-Mail=_85F6A1F8-FEF1-497E-9150-FB4AA6796345--