Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 78FCB9009 for ; Mon, 20 Feb 2012 01:00:51 +0000 (UTC) Received: (qmail 66701 invoked by uid 500); 20 Feb 2012 01:00:49 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 66677 invoked by uid 500); 20 Feb 2012 01:00:49 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 66667 invoked by uid 99); 20 Feb 2012 01:00:49 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Feb 2012 01:00:49 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=FSL_RCVD_USER,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [208.113.200.5] (HELO homiemail-a44.g.dreamhost.com) (208.113.200.5) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Feb 2012 01:00:41 +0000 Received: from homiemail-a44.g.dreamhost.com (localhost [127.0.0.1]) by homiemail-a44.g.dreamhost.com (Postfix) with ESMTP id 613E1118058 for ; Sun, 19 Feb 2012 17:00:20 -0800 (PST) DomainKey-Signature: a=rsa-sha1; c=nofws; d=thelastpickle.com; h=from :mime-version:content-type:subject:date:in-reply-to:to :references:message-id; q=dns; s=thelastpickle.com; b=Cwl4F1mb+o cPjhiIVeqj8RpM+4IrQ1BOR+MiQECUUBtBuPvN4LkDUU4ed2cl5KRNwawxdZYxQZ 2Lq+bY2GfzuYh6ynmmlgAgMHsXmlxPwaXmqYFAoH77ckcdCiuoilbsIX95TVBA2J 9sAlTnWLx9/JPH3UYkua5naBet7Hr95Nw= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=thelastpickle.com; h=from :mime-version:content-type:subject:date:in-reply-to:to :references:message-id; s=thelastpickle.com; bh=QkDGvehn2Y/4TmI/ EAB/ZD93zPA=; b=deSIEEaRBLofRsput/VZ1jCLH0Psyuu2LV8jLnPK2776imo3 BjMFRHhp+EqC8unIyGnxpwuiSdD3UieXKaVAswtEnzljDL/UDQ9CBeFBKfD6Ehfs oK5w3btZabAq1E47Ue6iMeZw5mAl0GrA7d3JwiwDYPyTMzjfkjp5wp2n95k= Received: from 202-126-206-153.vectorcommunications.net.nz (unknown [202.126.206.153]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: aaron@thelastpickle.com) by homiemail-a44.g.dreamhost.com (Postfix) with ESMTPSA id D06A3118057 for ; Sun, 19 Feb 2012 17:00:19 -0800 (PST) From: aaron morton Mime-Version: 1.0 (Apple Message framework v1251.1) Content-Type: multipart/alternative; boundary="Apple-Mail=_844D7FE6-14B8-4828-BCA3-882570048A44" Subject: Re: reads/s suddenly dropped Date: Mon, 20 Feb 2012 14:00:15 +1300 In-Reply-To: To: user@cassandra.apache.org References: Message-Id: X-Mailer: Apple Mail (2.1251.1) X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail=_844D7FE6-14B8-4828-BCA3-882570048A44 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=windows-1252 Aside from iostats.. nodetool cfstats will give you read and write latency for each CF. This = is the latency for the operation on each node. Check that to see if = latency is increasing.=20 Take a look at nodetool compactionstats to see if compactions are = running at the same time. The IO is throttled but if you are on aws it = may not be throttled enough.=20 The sweet spot for non netflix deployments seems to be a m1.xlarge with = 16GB. THe JVM can have 8 and the rest can be used for memmapping files. = Here is a good post about choosing EC2 sizes=85 = http://perfcap.blogspot.co.nz/2011/03/understanding-and-using-amazon-ebs.h= tml Cheers ----------------- Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 20/02/2012, at 9:31 AM, Franc Carter wrote: > On Mon, Feb 20, 2012 at 4:10 AM, Philippe wrote: > Perhaps your dataset can no longer be held in memory. Check iostats >=20 >=20 > I have been flushing the keycache and dropping the linux disk caches = before each to avoid testing memory reads. >=20 > One possibility that I thought of is that the success keys are now = 'far enough away' that they are not being included in the previous read = and hence the seek penalty has to be paid a lot more often - viable ? > =20 > cheers >=20 > Le 19 f=E9vr. 2012 11:24, "Franc Carter" a = =E9crit : >=20 >=20 > I've been testing Cassandra - primarily looking at reads/second for = our fairly data model - one unique key with a row of columns that we = always request. I've now setup the cluster with with m1.large (2 cpus = 8GB) >=20 > I had loaded a months worth of data in and was doing random requests = as a torture test - and getting very nice results. I then loaded another = days worth of day and repeated the tests while the load was running - = still good. >=20 > I then started loading more days and at some point the performance = dropped by close to an order of magnitude ;-( >=20 > Any ideas on what to look for ? >=20 > thanks >=20 > --=20 > Franc Carter | Systems architect | Sirca Ltd > franc.carter@sirca.org.au | www.sirca.org.au > Tel: +61 2 9236 9118=20 > Level 9, 80 Clarence St, Sydney NSW 2000 > PO Box H58, Australia Square, Sydney NSW 1215 >=20 >=20 >=20 >=20 > --=20 > Franc Carter | Systems architect | Sirca Ltd > franc.carter@sirca.org.au | www.sirca.org.au > Tel: +61 2 9236 9118=20 > Level 9, 80 Clarence St, Sydney NSW 2000 > PO Box H58, Australia Square, Sydney NSW 1215 >=20 --Apple-Mail=_844D7FE6-14B8-4828-BCA3-882570048A44 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=windows-1252 Aside = from iostats..

nodetool cfstats will give you read = and write latency for each CF. This is the latency for the operation on = each node. Check that to see if latency is = increasing. 

Take a look at nodetool = compactionstats to see if compactions are running at the same time. The = IO is throttled but if you are on aws it may not be throttled = enough. 

The sweet spot for non netflix = deployments seems to be a m1.xlarge with 16GB. THe JVM can have 8 and = the rest can be used for memmapping files. Here is a good post about = choosing EC2 sizes=85 http://perfcap.blogspot.co.nz/2011/03/understanding-and-using= -amazon-ebs.html

Cheers

http://www.thelastpickle.com

On 20/02/2012, at 9:31 AM, Franc Carter wrote:

On Mon, = Feb 20, 2012 at 4:10 AM, Philippe <watcherfr@gmail.com> = wrote:

Perhaps your dataset can no longer be held in = memory. Check iostats


I have been = flushing the keycache and dropping the linux disk caches before each to = avoid testing memory reads.

One possibility that I thought of is that the = success keys are now 'far enough away' that they are not being included = in the previous read and hence the seek penalty has to be paid a lot = more often  - viable ?
 
cheers

Le 19 f=E9vr. 2012 11:24, "Franc Carter" = <franc.carter@sirca.org.au> a = =E9crit :


I've been testing Cassandra - primarily looking at = reads/second for our fairly data model - one unique key with a row of = columns that we always request. I've now setup the cluster with with = m1.large (2 cpus 8GB)

I had loaded a months worth of data in and was doing = random requests as a torture test - and getting very nice results. I = then loaded another days worth of day and repeated the tests while the = load was running - still good.

I then started loading more days and at some point = the performance dropped by close to an order of magnitude = ;-(

Any ideas on what to look for = ?

thanks

--
Franc Carter | Systems architect | = Sirca = Ltd
Level 9, 80 Clarence St, = Sydney NSW 2000
PO Box H58, Australia Square, = Sydney NSW 1215




--
Franc Carter | Systems architect | = Sirca = Ltd
Level 9, 80 Clarence St, = Sydney NSW 2000
PO Box H58, Australia Square, = Sydney NSW 1215


= --Apple-Mail=_844D7FE6-14B8-4828-BCA3-882570048A44--